2024-03-11T22:42:47.390 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T22:42:47.391 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032 --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} -- /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/orig.config.yaml 2024-03-11T22:42:47.418 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:42:47.499 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032 branch: squid description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} email: ceph-qa@ceph.com first_in_suite: false job_id: '7592032' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 94a539707733325322964c6a187c6a1b41be1daf ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 94a539707733325322964c6a187c6a1b41be1daf selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 3176 sha1: 94a539707733325322964c6a187c6a1b41be1daf sleep_before_teardown: 0 subset: 14/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: fbcdfe674e56d1ea7a0810744e5915bb07961be5 targets: smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLSqpIXwqoo3HpDn9TD0OOJwViOUqXSLR4yiFNtWg4Kr2m4I23H49N2+5gXmUp+v6RaPTVUbM+uvFLGNfQTtr0Q= smithi131.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY2Rv8Cyuf2grkHYrJp+swE8nONHUG3IwxKX2IISZcByZzxRAZWcqRfUz4dJcBHKVZGYM/LiOZL72omdtxaiQs= smithi195.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJbFDz/gnTug6cnYTXefQ/S1iqAwKM7AOLnpzvSHd54VEHPsn8szxjYLrvDh229KOl1kWeyHZs2fjcy7BROhulM= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T22:42:47.499 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa; will attempt to use it 2024-03-11T22:42:47.499 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_fbcdfe674e56d1ea7a0810744e5915bb07961be5/qa/tasks 2024-03-11T22:42:47.500 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T22:42:47.501 INFO:teuthology.task.internal:Checking packages... 2024-03-11T22:42:47.527 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '94a539707733325322964c6a187c6a1b41be1daf' 2024-03-11T22:42:47.527 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T22:42:47.527 INFO:teuthology.packaging:ref: None 2024-03-11T22:42:47.527 INFO:teuthology.packaging:tag: None 2024-03-11T22:42:47.527 INFO:teuthology.packaging:branch: squid 2024-03-11T22:42:47.527 INFO:teuthology.packaging:sha1: 94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:42:47.527 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-11T22:42:47.678 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1607.g94a53970 2024-03-11T22:42:47.680 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T22:42:47.687 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T22:42:47.687 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T22:42:47.717 INFO:teuthology.task.internal:Saving configuration 2024-03-11T22:42:47.729 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T22:42:47.735 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:42:47.760 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:37:42.507855', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLSqpIXwqoo3HpDn9TD0OOJwViOUqXSLR4yiFNtWg4Kr2m4I23H49N2+5gXmUp+v6RaPTVUbM+uvFLGNfQTtr0Q='} 2024-03-11T22:42:47.789 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:37:42.506070', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY2Rv8Cyuf2grkHYrJp+swE8nONHUG3IwxKX2IISZcByZzxRAZWcqRfUz4dJcBHKVZGYM/LiOZL72omdtxaiQs='} 2024-03-11T22:42:47.814 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:37:42.508815', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJbFDz/gnTug6cnYTXefQ/S1iqAwKM7AOLnpzvSHd54VEHPsn8szxjYLrvDh229KOl1kWeyHZs2fjcy7BROhulM='} 2024-03-11T22:42:47.814 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T22:42:47.819 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T22:42:47.819 INFO:teuthology.task.internal:roles: ubuntu@smithi131.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-11T22:42:47.819 INFO:teuthology.task.internal:roles: ubuntu@smithi195.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-03-11T22:42:47.819 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T22:42:47.927 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f56c26c7f70>, signals=[15]) 2024-03-11T22:42:47.928 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T22:42:47.934 INFO:teuthology.task.internal:Opening connections... 2024-03-11T22:42:47.935 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-03-11T22:42:47.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:42:48.038 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi131.front.sepia.ceph.com 2024-03-11T22:42:48.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:42:48.155 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi195.front.sepia.ceph.com 2024-03-11T22:42:48.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:42:48.241 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T22:42:48.247 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-03-11T22:42:48.265 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-03-11T22:42:48.266 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:42:48.323 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:42:48.324 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:42:48.324 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:42:48.324 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:42:48.324 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-03-11T22:42:48.343 DEBUG:teuthology.orchestra.run.smithi131:> uname -m 2024-03-11T22:42:48.361 INFO:teuthology.orchestra.run.smithi131.stdout:x86_64 2024-03-11T22:42:48.361 DEBUG:teuthology.orchestra.run.smithi131:> cat /etc/os-release 2024-03-11T22:42:48.419 INFO:teuthology.orchestra.run.smithi131.stdout:NAME="CentOS Stream" 2024-03-11T22:42:48.419 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION="9" 2024-03-11T22:42:48.419 INFO:teuthology.orchestra.run.smithi131.stdout:ID="centos" 2024-03-11T22:42:48.419 INFO:teuthology.orchestra.run.smithi131.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:42:48.419 INFO:teuthology.orchestra.run.smithi131.stdout:VERSION_ID="9" 2024-03-11T22:42:48.419 INFO:teuthology.orchestra.run.smithi131.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:ANSI_COLOR="0;31" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:42:48.420 INFO:teuthology.orchestra.run.smithi131.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:42:48.420 INFO:teuthology.lock.ops:Updating smithi131.front.sepia.ceph.com on lock server 2024-03-11T22:42:48.443 DEBUG:teuthology.orchestra.run.smithi195:> uname -m 2024-03-11T22:42:48.461 INFO:teuthology.orchestra.run.smithi195.stdout:x86_64 2024-03-11T22:42:48.461 DEBUG:teuthology.orchestra.run.smithi195:> cat /etc/os-release 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:NAME="CentOS Stream" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION="9" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:ID="centos" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:ID_LIKE="rhel fedora" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:VERSION_ID="9" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:PLATFORM_ID="platform:el9" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:ANSI_COLOR="0;31" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:LOGO="fedora-logo-icon" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:HOME_URL="https://centos.org/" 2024-03-11T22:42:48.518 INFO:teuthology.orchestra.run.smithi195.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T22:42:48.519 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T22:42:48.519 INFO:teuthology.orchestra.run.smithi195.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T22:42:48.519 INFO:teuthology.lock.ops:Updating smithi195.front.sepia.ceph.com on lock server 2024-03-11T22:42:48.543 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T22:42:48.553 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T22:42:48.559 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T22:42:48.559 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:42:48.562 DEBUG:teuthology.orchestra.run.smithi131:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:42:48.564 DEBUG:teuthology.orchestra.run.smithi195:> test '!' -e /home/ubuntu/cephtest 2024-03-11T22:42:48.580 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T22:42:48.586 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T22:42:48.586 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:42:48.619 DEBUG:teuthology.orchestra.run.smithi131:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:42:48.621 DEBUG:teuthology.orchestra.run.smithi195:> test -z $(ls -A /var/lib/ceph) 2024-03-11T22:42:48.709 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T22:42:48.782 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T22:42:48.796 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T22:42:48.797 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T22:42:48.797 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:42:48.797 DEBUG:teuthology.orchestra.run.smithi131:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:42:48.798 DEBUG:teuthology.orchestra.run.smithi195:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T22:42:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:42:48.814 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-03-11T22:42:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:42:48.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:42:48.816 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-03-11T22:42:48.816 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-03-11T22:42:48.872 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:42:48.872 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-361.el9.x86_64 2024-03-11T22:42:48.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-03-11T22:42:48.875 INFO:teuthology.orchestra.run.smithi131.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:42:48.876 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-361.el9.x86_64 2024-03-11T22:42:48.876 INFO:teuthology.task.kernel:Running kernel on smithi131: 5.14.0-361.el9.x86_64 2024-03-11T22:42:48.876 DEBUG:teuthology.orchestra.run.smithi131:> sudo yum install -y kernel 2024-03-11T22:42:48.877 INFO:teuthology.task.kernel:Running kernel on smithi195: 5.14.0-361.el9.x86_64 2024-03-11T22:42:48.877 DEBUG:teuthology.orchestra.run.smithi195:> sudo yum install -y kernel 2024-03-11T22:42:50.953 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-11T22:42:50.994 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T22:42:50.996 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T22:42:52.596 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-11T22:42:52.612 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-03-11T22:42:52.746 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-9 - BaseOS 3.1 MB/s | 2.0 MB 00:00 2024-03-11T22:42:53.814 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-11T22:42:55.721 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-11T22:43:02.185 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - BaseOS 889 kB/s | 8.0 MB 00:09 2024-03-11T22:43:05.990 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 919 kB/s | 6.1 MB 00:06 2024-03-11T22:43:06.225 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - BaseOS 619 kB/s | 8.0 MB 00:13 2024-03-11T22:43:07.396 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 15 kB 00:00 2024-03-11T22:43:08.477 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 19 MB 00:05 2024-03-11T22:43:09.201 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - AppStream 10 MB/s | 19 MB 00:01 2024-03-11T22:43:10.239 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 7.5 MB/s | 21 MB 00:02 2024-03-11T22:43:12.486 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-11T22:43:14.438 INFO:teuthology.orchestra.run.smithi131.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 15 kB 00:00 2024-03-11T22:43:15.221 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-11T22:43:17.361 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:43:17.361 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:43:17.455 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:43:17.462 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-11T22:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-11T22:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-03-11T22:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 100 M 2024-03-11T22:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 123 M 2024-03-11T22:43:17.463 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-11T22:43:24.034 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - CRB 551 kB/s | 6.1 MB 00:11 2024-03-11T22:43:26.843 INFO:teuthology.orchestra.run.smithi195.stdout:CentOS Stream 9 - Extras packages 9.2 kB/s | 15 kB 00:01 2024-03-11T22:43:29.263 INFO:teuthology.orchestra.run.smithi195.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 21 MB 00:02 2024-03-11T22:43:30.281 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 480 kB/s | 6.0 MB 00:12 2024-03-11T22:43:34.135 INFO:teuthology.orchestra.run.smithi195.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-11T22:43:36.309 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:43:36.309 INFO:teuthology.orchestra.run.smithi195.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:43:36.406 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-11T22:43:36.413 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-11T22:43:36.413 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-03-11T22:43:36.413 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout:Installing dependencies: 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout:Install 4 Packages 2024-03-11T22:43:36.414 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:43:36.415 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 100 M 2024-03-11T22:43:36.415 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 123 M 2024-03-11T22:43:36.415 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-11T22:43:37.745 INFO:teuthology.orchestra.run.smithi195.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 5.4 MB/s | 6.0 MB 00:01 2024-03-11T22:43:38.830 INFO:teuthology.orchestra.run.smithi131.stdout:Extra Packages for Enterprise Linux 868 kB/s | 21 MB 00:24 2024-03-11T22:43:42.854 INFO:teuthology.orchestra.run.smithi195.stdout:(2/4): kernel-modules-core-5.14.0-427.el9.x86_6 6.5 MB/s | 33 MB 00:05 2024-03-11T22:43:43.162 INFO:teuthology.orchestra.run.smithi195.stdout:(3/4): kernel-core-5.14.0-427.el9.x86_64.rpm 3.2 MB/s | 21 MB 00:06 2024-03-11T22:43:43.752 INFO:teuthology.orchestra.run.smithi131.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-11T22:43:45.379 INFO:teuthology.orchestra.run.smithi195.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 4.5 MB/s | 39 MB 00:08 2024-03-11T22:43:45.379 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:43:45.379 INFO:teuthology.orchestra.run.smithi195.stdout:Total 11 MB/s | 100 MB 00:08 2024-03-11T22:43:45.942 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T22:43:45.943 INFO:teuthology.orchestra.run.smithi131.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T22:43:46.029 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout:Installing dependencies: 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T22:43:46.039 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T22:43:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:43:46.040 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-03-11T22:43:46.040 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-03-11T22:43:46.040 INFO:teuthology.orchestra.run.smithi131.stdout:Install 4 Packages 2024-03-11T22:43:46.040 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:43:46.040 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 100 M 2024-03-11T22:43:46.041 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 123 M 2024-03-11T22:43:46.041 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-03-11T22:43:46.268 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-11T22:43:46.362 INFO:teuthology.orchestra.run.smithi131.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 21 MB/s | 6.0 MB 00:00 2024-03-11T22:43:46.640 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-11T22:43:46.641 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-11T22:43:46.696 INFO:teuthology.orchestra.run.smithi131.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 34 MB/s | 21 MB 00:00 2024-03-11T22:43:47.296 INFO:teuthology.orchestra.run.smithi131.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 32 MB/s | 39 MB 00:01 2024-03-11T22:43:47.622 INFO:teuthology.orchestra.run.smithi131.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 27 MB/s | 33 MB 00:01 2024-03-11T22:43:47.622 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:43:47.622 INFO:teuthology.orchestra.run.smithi131.stdout:Total 63 MB/s | 100 MB 00:01 2024-03-11T22:43:48.002 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-11T22:43:48.003 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-11T22:43:48.507 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-03-11T22:43:48.898 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-03-11T22:43:48.899 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-03-11T22:43:50.218 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-03-11T22:43:50.219 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-03-11T22:43:50.906 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-11T22:43:51.548 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:43:51.679 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:43:52.924 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-03-11T22:43:53.570 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:43:53.709 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:43:54.448 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:43:54.622 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:43:54.975 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 570 kB/s | 21 MB 00:37 2024-03-11T22:43:56.472 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:43:56.655 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:43:59.355 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:43:59.576 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:01.360 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:44:01.498 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:04.308 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:06.257 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:24.144 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 635 kB/s | 33 MB 00:53 2024-03-11T22:44:30.843 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 549 kB/s | 39 MB 01:13 2024-03-11T22:44:30.843 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:44:30.843 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.4 MB/s | 100 MB 01:13 2024-03-11T22:44:31.723 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-11T22:44:32.095 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-11T22:44:32.095 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-11T22:44:33.462 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:33.495 INFO:teuthology.orchestra.run.smithi195.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:44:33.495 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:44:33.546 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:33.739 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-11T22:44:33.739 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-11T22:44:33.988 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:33.988 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:44:33.988 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:44:33.988 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:44:34.383 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:34.384 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:44:34.384 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-11T22:44:34.384 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:44:34.384 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:44:34.384 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:44:34.385 INFO:teuthology.orchestra.run.smithi195.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:44:34.385 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:44:34.385 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-11T22:44:34.882 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:44:35.026 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:44:35.027 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:44:35.027 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:44:35.027 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:44:35.027 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:44:35.027 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-03-11T22:44:35.027 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi195.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:44:35.027 DEBUG:teuthology.orchestra.run.smithi195:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:44:35.148 INFO:teuthology.orchestra.run.smithi195.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:44:35.148 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:44:35.148 DEBUG:teuthology.orchestra.run.smithi195:> sudo rpm -qi grub2-tools 2024-03-11T22:44:35.182 INFO:teuthology.orchestra.run.smithi195.stdout:Name : grub2-tools 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Epoch : 1 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Version : 2.06 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Release : 68.el9 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Architecture: x86_64 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Group : Unspecified 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Size : 8264254 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:License : GPLv3+ 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:44:35.183 INFO:teuthology.orchestra.run.smithi195.stdout:Packager : builder@centos.org 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:Vendor : CentOS 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:Summary : Support tools for GRUB. 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:Description : 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:hardware devices. 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:44:35.184 INFO:teuthology.orchestra.run.smithi195.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:44:35.185 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:44:35.185 INFO:teuthology.task.kernel:Updating grub on smithi195 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:44:35.186 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:44:35.575 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:35.575 INFO:teuthology.orchestra.run.smithi131.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:44:35.575 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:44:35.659 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:35.796 INFO:teuthology.orchestra.run.smithi195.stderr:Generating grub configuration file ... 2024-03-11T22:44:36.120 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:36.120 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:44:36.120 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:44:36.120 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:44:36.445 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-11T22:44:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:44:36.551 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-03-11T22:44:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:44:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:44:36.551 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:44:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:44:36.552 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:44:36.552 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-03-11T22:44:37.087 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:44:37.135 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:44:37.166 INFO:teuthology.orchestra.run.smithi195.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:44:37.203 INFO:teuthology.orchestra.run.smithi195.stderr:done 2024-03-11T22:44:37.205 DEBUG:teuthology.orchestra.run.smithi195:> mktemp 2024-03-11T22:44:37.218 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:44:37.220 INFO:teuthology.orchestra.run.smithi195.stdout:/tmp/tmp.OWoixYyU45 2024-03-11T22:44:37.220 DEBUG:teuthology.orchestra.run.smithi195:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.OWoixYyU45 2024-03-11T22:44:37.270 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:44:37.271 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:44:37.271 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:44:37.271 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:44:37.271 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:44:37.271 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T22:44:37.271 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi131.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:44:37.271 DEBUG:teuthology.orchestra.run.smithi131:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:44:37.289 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0666 /tmp/tmp.OWoixYyU45 2024-03-11T22:44:37.392 INFO:teuthology.orchestra.run.smithi131.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:44:37.392 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:44:37.392 DEBUG:teuthology.orchestra.run.smithi131:> sudo rpm -qi grub2-tools 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Name : grub2-tools 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Epoch : 1 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Version : 2.06 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Release : 68.el9 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Architecture: x86_64 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Group : Unspecified 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Size : 8264254 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:License : GPLv3+ 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:44:37.427 INFO:teuthology.orchestra.run.smithi131.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:Packager : builder@centos.org 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:Vendor : CentOS 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:Summary : Support tools for GRUB. 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:Description : 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:hardware devices. 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:44:37.428 INFO:teuthology.orchestra.run.smithi131.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:44:37.429 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:44:37.429 INFO:teuthology.task.kernel:Updating grub on smithi131 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:44:37.429 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:44:37.442 DEBUG:teuthology.orchestra.remote:smithi195:/tmp/tmp.OWoixYyU45 is 6KB 2024-03-11T22:44:37.491 DEBUG:teuthology.orchestra.run.smithi195:> rm -fr /tmp/tmp.OWoixYyU45 2024-03-11T22:44:37.505 DEBUG:teuthology.orchestra.run.smithi195:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:44:37.578 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:44:37.579 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:44:37.579 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:44:37.579 INFO:teuthology.orchestra.run.smithi195.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:44:37.580 DEBUG:teuthology.orchestra.run.smithi195:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:44:37.747 DEBUG:teuthology.orchestra.run.smithi195:> sudo shutdown -r now 2024-03-11T22:44:38.008 INFO:teuthology.orchestra.run.smithi131.stderr:Generating grub configuration file ... 2024-03-11T22:44:39.438 INFO:teuthology.orchestra.run.smithi131.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:44:39.458 INFO:teuthology.orchestra.run.smithi131.stderr:done 2024-03-11T22:44:39.460 DEBUG:teuthology.orchestra.run.smithi131:> mktemp 2024-03-11T22:44:39.474 INFO:teuthology.orchestra.run.smithi131.stdout:/tmp/tmp.h5hPCjwUsK 2024-03-11T22:44:39.474 DEBUG:teuthology.orchestra.run.smithi131:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.h5hPCjwUsK 2024-03-11T22:44:39.543 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0666 /tmp/tmp.h5hPCjwUsK 2024-03-11T22:44:39.677 DEBUG:teuthology.orchestra.remote:smithi131:/tmp/tmp.h5hPCjwUsK is 6KB 2024-03-11T22:44:39.727 DEBUG:teuthology.orchestra.run.smithi131:> rm -fr /tmp/tmp.h5hPCjwUsK 2024-03-11T22:44:39.741 DEBUG:teuthology.orchestra.run.smithi131:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:44:39.806 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:44:39.806 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:44:39.806 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:44:39.806 INFO:teuthology.orchestra.run.smithi131.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:44:39.807 DEBUG:teuthology.orchestra.run.smithi131:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:44:39.956 DEBUG:teuthology.orchestra.run.smithi131:> sudo shutdown -r now 2024-03-11T22:44:39.983 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:44:40.161 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:44:44.908 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:44:45.104 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:44:49.883 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:45:07.750 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-03-11T22:45:07.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-11T22:45:07.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:45:09.959 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T22:45:09.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-11T22:45:09.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:45:19.194 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:45:19.194 INFO:teuthology.orchestra.run.smithi107.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T22:45:19.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:45:19.281 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:45:19.689 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:45:19.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T22:45:19.689 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T22:45:19.690 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T22:45:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T22:45:20.121 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:45:20.122 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-11T22:45:20.122 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T22:45:20.122 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T22:45:20.122 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T22:45:20.122 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T22:45:20.122 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:45:20.122 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-11T22:45:20.663 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:45:20.804 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:45:20.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:45:20.804 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T22:45:20.804 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T22:45:20.804 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T22:45:20.805 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T22:45:20.805 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T22:45:20.805 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T22:45:20.925 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T22:45:20.925 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T22:45:20.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-03-11T22:45:20.965 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T22:45:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:45:20.967 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T22:45:20.968 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T22:45:20.968 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-427.el9.x86_64 2024-03-11T22:45:20.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T22:45:21.594 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-03-11T22:45:22.995 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T22:45:23.015 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-03-11T22:45:23.017 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-03-11T22:45:23.033 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.XfPsUxnrrb 2024-03-11T22:45:23.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XfPsUxnrrb 2024-03-11T22:45:23.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.XfPsUxnrrb 2024-03-11T22:45:23.244 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.XfPsUxnrrb is 6KB 2024-03-11T22:45:23.293 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.XfPsUxnrrb 2024-03-11T22:45:23.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T22:45:23.377 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T22:45:23.377 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T22:45:23.377 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T22:45:23.377 INFO:teuthology.orchestra.run.smithi107.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T22:45:23.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T22:45:23.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-03-11T22:45:26.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.195 2024-03-11T22:45:28.479 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.131 2024-03-11T22:45:35.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-11T22:45:35.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:45:37.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-11T22:45:37.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:45:53.550 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T22:45:53.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-11T22:45:53.552 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:46:11.999 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-11T22:46:21.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-11T22:46:21.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:46:24.066 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-11T22:46:35.194 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:46:36.068 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-11T22:46:36.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:46:37.486 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T22:46:47.203 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-11T22:46:47.204 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:46:47.584 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-03-11T22:46:47.975 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-11T22:46:47.975 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:46:47.975 DEBUG:teuthology.orchestra.run.smithi195:> uname -r 2024-03-11T22:46:47.990 INFO:teuthology.orchestra.run.smithi195.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:46:47.990 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:46:47.990 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:46:47.990 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:46:48.991 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-03-11T22:46:48.991 DEBUG:teuthology.orchestra.run.smithi195:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:46:49.068 INFO:teuthology.orchestra.run.smithi195.stdout:ttyS1 2024-03-11T22:46:49.101 DEBUG:teuthology.parallel:result is None 2024-03-11T22:46:49.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-11T22:46:49.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:46:49.844 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-03-11T22:46:50.226 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-11T22:46:50.226 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:46:50.226 DEBUG:teuthology.orchestra.run.smithi131:> uname -r 2024-03-11T22:46:50.242 INFO:teuthology.orchestra.run.smithi131.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:46:50.242 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:46:50.242 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:46:50.242 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:46:51.243 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T22:46:51.243 DEBUG:teuthology.orchestra.run.smithi131:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:46:51.322 INFO:teuthology.orchestra.run.smithi131.stdout:ttyS1 2024-03-11T22:46:51.355 DEBUG:teuthology.parallel:result is None 2024-03-11T22:46:54.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-03-11T22:47:09.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-11T22:47:09.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:47:09.938 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-11T22:47:10.363 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-11T22:47:10.363 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T22:47:10.363 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-03-11T22:47:10.377 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-427.el9.x86_64 2024-03-11T22:47:10.378 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T22:47:10.378 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T22:47:10.378 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T22:47:11.379 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T22:47:11.379 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T22:47:11.450 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-03-11T22:47:11.492 DEBUG:teuthology.parallel:result is None 2024-03-11T22:47:11.492 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T22:47:11.500 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T22:47:11.500 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:47:11.502 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:47:11.504 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T22:47:11.522 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T22:47:11.529 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T22:47:11.536 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T22:47:11.536 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:47:11.561 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:47:11.564 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T22:47:11.620 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T22:47:11.627 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T22:47:11.628 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:47:11.631 DEBUG:teuthology.orchestra.run.smithi131:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:47:11.633 DEBUG:teuthology.orchestra.run.smithi195:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T22:47:11.658 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:47:11.670 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:47:11.671 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:47:11.684 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:47:11.700 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:47:11.714 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T22:47:11.716 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T22:47:11.723 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T22:47:11.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:47:11.725 DEBUG:teuthology.orchestra.run.smithi131:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:47:11.728 DEBUG:teuthology.orchestra.run.smithi195:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T22:47:11.757 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T22:47:11.767 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T22:47:11.767 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:47:11.797 DEBUG:teuthology.orchestra.run.smithi131:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:47:11.799 DEBUG:teuthology.orchestra.run.smithi195:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T22:47:11.817 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:47:11.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:47:11.950 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:47:12.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:47:12.082 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:47:12.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:47:12.152 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:47:12.196 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:47:12.254 DEBUG:teuthology.orchestra.run.smithi131:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:47:12.335 DEBUG:teuthology.orchestra.run.smithi131:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:47:12.381 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:47:12.381 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:47:12.450 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:47:12.496 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T22:47:12.554 DEBUG:teuthology.orchestra.run.smithi195:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:47:12.640 DEBUG:teuthology.orchestra.run.smithi195:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T22:47:12.685 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:47:12.685 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T22:47:12.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-03-11T22:47:12.759 DEBUG:teuthology.orchestra.run.smithi131:> sudo service rsyslog restart 2024-03-11T22:47:12.761 DEBUG:teuthology.orchestra.run.smithi195:> sudo service rsyslog restart 2024-03-11T22:47:12.805 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:47:12.834 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:47:12.839 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:47:13.209 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T22:47:13.219 INFO:teuthology.task.internal:Starting timer... 2024-03-11T22:47:13.219 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T22:47:13.244 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T22:47:13.253 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T22:47:13.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-03-11T22:47:13.330 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-03-11T22:47:13.335 DEBUG:teuthology.orchestra.run.smithi131:> sudo service auditd rotate 2024-03-11T22:47:13.394 INFO:teuthology.orchestra.run.smithi131.stdout:Rotating logs: 2024-03-11T22:47:13.396 DEBUG:teuthology.orchestra.run.smithi195:> sudo service auditd rotate 2024-03-11T22:47:13.455 INFO:teuthology.orchestra.run.smithi195.stdout:Rotating logs: 2024-03-11T22:47:13.457 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T22:47:13.457 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-03-11T22:47:13.499 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-03-11T22:47:13.499 DEBUG:teuthology.orchestra.run.smithi131:> /usr/sbin/getenforce 2024-03-11T22:47:13.521 INFO:teuthology.orchestra.run.smithi131.stdout:Permissive 2024-03-11T22:47:13.521 DEBUG:teuthology.orchestra.run.smithi195:> /usr/sbin/getenforce 2024-03-11T22:47:13.542 INFO:teuthology.orchestra.run.smithi195.stdout:Permissive 2024-03-11T22:47:13.542 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi131.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi195.front.sepia.ceph.com': 'permissive'} 2024-03-11T22:47:13.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:47:13.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:47:13.597 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:47:13.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:47:13.625 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:47:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:47:13.654 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T22:47:13.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:47:13.684 DEBUG:teuthology.orchestra.run.smithi131:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:47:13.722 DEBUG:teuthology.orchestra.run.smithi195:> sudo /usr/sbin/setenforce permissive 2024-03-11T22:47:13.752 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T22:47:13.763 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-11T22:47:13.768 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-11T22:47:13.894 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T22:47:13.914 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T22:47:13.916 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com,smithi131.front.sepia.ceph.com,smithi195.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T22:51:25.058 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi131.front.sepia.ceph.com'), Remote(name='ubuntu@smithi195.front.sepia.ceph.com')] 2024-03-11T22:51:25.060 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-11T22:51:25.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:51:25.163 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-03-11T22:51:25.242 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-03-11T22:51:25.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-11T22:51:25.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi131.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:51:25.330 DEBUG:teuthology.orchestra.run.smithi131:> true 2024-03-11T22:51:25.417 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi131.front.sepia.ceph.com' 2024-03-11T22:51:25.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-11T22:51:25.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi195.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T22:51:25.505 DEBUG:teuthology.orchestra.run.smithi195:> true 2024-03-11T22:51:25.582 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi195.front.sepia.ceph.com' 2024-03-11T22:51:25.582 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T22:51:25.593 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T22:51:25.594 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:51:25.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:51:25.596 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:51:25.597 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:51:25.599 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T22:51:25.599 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:51:25.626 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:51:25.628 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:51:25.643 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:51:25.645 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:51:25.654 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T22:51:25.671 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T22:51:25.672 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-03-11T22:51:25.675 INFO:teuthology.orchestra.run.smithi131.stderr:sudo: ntpd: command not found 2024-03-11T22:51:25.686 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-03-11T22:51:25.689 INFO:teuthology.orchestra.run.smithi131.stdout:506 Cannot talk to daemon 2024-03-11T22:51:25.700 INFO:teuthology.orchestra.run.smithi195.stderr:sudo: ntpd: command not found 2024-03-11T22:51:25.703 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:51:25.705 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:51:25.714 INFO:teuthology.orchestra.run.smithi195.stdout:506 Cannot talk to daemon 2024-03-11T22:51:25.719 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:51:25.721 INFO:teuthology.orchestra.run.smithi131.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:51:25.730 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T22:51:25.746 INFO:teuthology.orchestra.run.smithi195.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T22:51:25.771 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:51:25.774 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:51:25.774 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-03-11T22:51:25.774 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.774 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.774 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.774 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.775 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:51:25.778 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:51:25.778 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-11T22:51:25.779 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.779 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.779 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.779 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.794 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:51:25.798 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:51:25.798 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-03-11T22:51:25.798 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.798 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.798 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.798 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:51:25.798 INFO:teuthology.run_tasks:Running task pexec... 2024-03-11T22:51:25.808 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-11T22:51:25.808 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi107.front.sepia.ceph.com 2024-03-11T22:51:25.809 DEBUG:teuthology.orchestra.run.smithi107:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:51:25.809 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi131.front.sepia.ceph.com 2024-03-11T22:51:25.809 DEBUG:teuthology.orchestra.run.smithi131:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:51:25.809 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi195.front.sepia.ceph.com 2024-03-11T22:51:25.809 DEBUG:teuthology.orchestra.run.smithi195:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-11T22:51:26.393 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:49 ago on Mon 11 Mar 2024 10:50:37 PM UTC. 2024-03-11T22:51:26.401 INFO:teuthology.orchestra.run.smithi131.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 10:50:38 PM UTC. 2024-03-11T22:51:26.421 INFO:teuthology.orchestra.run.smithi195.stdout:Last metadata expiration check: 0:00:48 ago on Mon 11 Mar 2024 10:50:38 PM UTC. 2024-03-11T22:51:26.539 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-03-11T22:51:26.548 INFO:teuthology.orchestra.run.smithi131.stdout:Dependencies resolved. 2024-03-11T22:51:26.549 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-03-11T22:51:26.549 INFO:teuthology.orchestra.run.smithi131.stdout: Package Architecture Version Repository Size 2024-03-11T22:51:26.549 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-03-11T22:51:26.549 INFO:teuthology.orchestra.run.smithi131.stdout:Installing: 2024-03-11T22:51:26.549 INFO:teuthology.orchestra.run.smithi131.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:51:26.549 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:51:26.549 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction Summary 2024-03-11T22:51:26.550 INFO:teuthology.orchestra.run.smithi131.stdout:================================================================================ 2024-03-11T22:51:26.550 INFO:teuthology.orchestra.run.smithi131.stdout:Install 1 Package 2024-03-11T22:51:26.550 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-03-11T22:51:26.552 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:51:26.560 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.1 M 2024-03-11T22:51:26.560 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 9.8 M 2024-03-11T22:51:26.560 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-03-11T22:51:26.565 INFO:teuthology.orchestra.run.smithi131.stdout:Total download size: 3.1 M 2024-03-11T22:51:26.565 INFO:teuthology.orchestra.run.smithi131.stdout:Installed size: 9.8 M 2024-03-11T22:51:26.565 INFO:teuthology.orchestra.run.smithi131.stdout:Downloading Packages: 2024-03-11T22:51:26.567 INFO:teuthology.orchestra.run.smithi195.stdout:Dependencies resolved. 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout: Package Architecture Version Repository Size 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout:Installing: 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction Summary 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout:================================================================================ 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout:Install 1 Package 2024-03-11T22:51:26.580 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:51:26.585 INFO:teuthology.orchestra.run.smithi195.stdout:Total download size: 3.1 M 2024-03-11T22:51:26.585 INFO:teuthology.orchestra.run.smithi195.stdout:Installed size: 9.8 M 2024-03-11T22:51:26.585 INFO:teuthology.orchestra.run.smithi195.stdout:Downloading Packages: 2024-03-11T22:51:27.540 INFO:teuthology.orchestra.run.smithi107.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.3 MB/s | 3.1 MB 00:00 2024-03-11T22:51:27.540 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:27.540 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.2 MB/s | 3.1 MB 00:00 2024-03-11T22:51:27.571 INFO:teuthology.orchestra.run.smithi131.stdout:runc-1.1.12-2.el9.x86_64.rpm 4.3 MB/s | 3.1 MB 00:00 2024-03-11T22:51:27.571 INFO:teuthology.orchestra.run.smithi131.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:27.572 INFO:teuthology.orchestra.run.smithi131.stdout:Total 3.1 MB/s | 3.1 MB 00:01 2024-03-11T22:51:27.575 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-03-11T22:51:27.581 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-03-11T22:51:27.581 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-03-11T22:51:27.608 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction check 2024-03-11T22:51:27.614 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction check succeeded. 2024-03-11T22:51:27.614 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction test 2024-03-11T22:51:27.661 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-03-11T22:51:27.661 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-03-11T22:51:27.694 INFO:teuthology.orchestra.run.smithi131.stdout:Transaction test succeeded. 2024-03-11T22:51:27.694 INFO:teuthology.orchestra.run.smithi131.stdout:Running transaction 2024-03-11T22:51:27.881 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-03-11T22:51:27.919 INFO:teuthology.orchestra.run.smithi131.stdout: Preparing : 1/1 2024-03-11T22:51:27.976 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:28.008 INFO:teuthology.orchestra.run.smithi131.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:28.519 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:28.613 INFO:teuthology.orchestra.run.smithi131.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:28.865 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:28.865 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:51:28.865 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-03-11T22:51:28.865 INFO:teuthology.orchestra.run.smithi107.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:51:28.865 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:51:28.865 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-03-11T22:51:28.927 INFO:teuthology.orchestra.run.smithi131.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:28.927 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:51:28.927 INFO:teuthology.orchestra.run.smithi131.stdout:Installed: 2024-03-11T22:51:28.927 INFO:teuthology.orchestra.run.smithi131.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:51:28.927 INFO:teuthology.orchestra.run.smithi131.stdout: 2024-03-11T22:51:28.927 INFO:teuthology.orchestra.run.smithi131.stdout:Complete! 2024-03-11T22:51:29.094 DEBUG:teuthology.parallel:result is None 2024-03-11T22:51:29.214 DEBUG:teuthology.parallel:result is None 2024-03-11T22:51:32.165 INFO:teuthology.orchestra.run.smithi195.stdout:runc-1.1.12-2.el9.x86_64.rpm 580 kB/s | 3.1 MB 00:05 2024-03-11T22:51:32.165 INFO:teuthology.orchestra.run.smithi195.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:51:32.165 INFO:teuthology.orchestra.run.smithi195.stdout:Total 574 kB/s | 3.1 MB 00:05 2024-03-11T22:51:32.203 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction check 2024-03-11T22:51:32.209 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction check succeeded. 2024-03-11T22:51:32.209 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction test 2024-03-11T22:51:32.291 INFO:teuthology.orchestra.run.smithi195.stdout:Transaction test succeeded. 2024-03-11T22:51:32.292 INFO:teuthology.orchestra.run.smithi195.stdout:Running transaction 2024-03-11T22:51:32.514 INFO:teuthology.orchestra.run.smithi195.stdout: Preparing : 1/1 2024-03-11T22:51:32.626 INFO:teuthology.orchestra.run.smithi195.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:33.209 INFO:teuthology.orchestra.run.smithi195.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:33.551 INFO:teuthology.orchestra.run.smithi195.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-11T22:51:33.551 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:51:33.551 INFO:teuthology.orchestra.run.smithi195.stdout:Installed: 2024-03-11T22:51:33.551 INFO:teuthology.orchestra.run.smithi195.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-11T22:51:33.551 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:51:33.551 INFO:teuthology.orchestra.run.smithi195.stdout:Complete! 2024-03-11T22:51:33.764 DEBUG:teuthology.parallel:result is None 2024-03-11T22:51:33.764 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-03-11T22:51:33.774 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-03-11T22:51:33.775 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:51:33.775 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:51:33.795 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:51:33.795 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-03-11T22:51:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:51:33.851 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-11T22:51:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:51:24.100249425 +0000 2024-03-11T22:51:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:01.918628314 +0000 2024-03-11T22:51:33.852 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:01.918628314 +0000 2024-03-11T22:51:33.852 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:01.918628314 +0000 2024-03-11T22:51:33.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:51:33.919 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:51:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:51:33.920 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.0002162 s, 2.4 MB/s 2024-03-11T22:51:33.921 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:51:33.978 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-03-11T22:51:34.035 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:51:34.035 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:34.035 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T22:51:34.035 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:34.035 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:34.035 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:51:24.100249425 +0000 2024-03-11T22:51:34.036 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:02.261622455 +0000 2024-03-11T22:51:34.036 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:02.261622455 +0000 2024-03-11T22:51:34.036 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:02.261622455 +0000 2024-03-11T22:51:34.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:51:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:51:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:51:34.102 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000210269 s, 2.4 MB/s 2024-03-11T22:51:34.103 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:51:34.160 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-03-11T22:51:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:51:34.215 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:34.215 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-11T22:51:34.215 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:34.215 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:34.215 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:51:24.100249425 +0000 2024-03-11T22:51:34.215 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:02.606616562 +0000 2024-03-11T22:51:34.216 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:02.606616562 +0000 2024-03-11T22:51:34.216 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:02.606616562 +0000 2024-03-11T22:51:34.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:51:34.281 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:51:34.282 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:51:34.282 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000192617 s, 2.7 MB/s 2024-03-11T22:51:34.283 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:51:34.339 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:51:24.100249425 +0000 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:02.949610703 +0000 2024-03-11T22:51:34.394 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:02.949610703 +0000 2024-03-11T22:51:34.395 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:02.949610703 +0000 2024-03-11T22:51:34.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:51:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:51:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:51:34.459 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000174799 s, 2.9 MB/s 2024-03-11T22:51:34.461 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:51:34.516 DEBUG:teuthology.orchestra.run.smithi107:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:51:34.649 INFO:teuthology.orchestra.run.smithi107.stdout:loop 2024-03-11T22:51:34.651 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_1... 2024-03-11T22:51:34.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:51:34.688 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-03-11T22:51:34.725 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:51:34.761 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_2... 2024-03-11T22:51:34.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:51:34.843 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-03-11T22:51:34.882 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:51:34.912 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_3... 2024-03-11T22:51:34.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:51:34.991 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-03-11T22:51:35.029 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:51:35.059 INFO:tasks.nvme_loop:Connecting nvme_loop smithi107:/dev/vg_nvme/lv_4... 2024-03-11T22:51:35.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:51:35.138 INFO:teuthology.orchestra.run.smithi107.stdout:1 2024-03-11T22:51:35.177 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:51:35.207 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:51:35.207 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:51:35.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo nvme list 2024-03-11T22:51:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:51:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:51:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme0n1 PHFT620400NK400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T22:51:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme1n1 56673ee7cf6493dbf5f3 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme2n1 4ee21345192a9b55fa81 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme3n1 6ad205e36222af176688 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:35.329 INFO:teuthology.orchestra.run.smithi107.stdout:/dev/nvme4n1 1e0d947f7373791b0782 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:35.330 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:51:35.330 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:51:35.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/scratch_devs 2024-03-11T22:51:35.397 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:51:35.397 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:51:35.416 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:51:35.416 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_1 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:51:24.060194719 +0000 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:02.963665800 +0000 2024-03-11T22:51:35.473 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:02.963665800 +0000 2024-03-11T22:51:35.474 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:02.963665800 +0000 2024-03-11T22:51:35.474 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:51:35.544 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:51:35.544 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:51:35.544 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000347821 s, 1.5 MB/s 2024-03-11T22:51:35.545 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:51:35.604 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_2 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:51:24.060194719 +0000 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:03.301658253 +0000 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:03.301658253 +0000 2024-03-11T22:51:35.661 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:03.301658253 +0000 2024-03-11T22:51:35.662 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:51:35.728 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:51:35.729 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:51:35.729 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000190836 s, 2.7 MB/s 2024-03-11T22:51:35.730 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:51:35.787 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_3 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:51:24.060194719 +0000 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:03.646650549 +0000 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:03.646650549 +0000 2024-03-11T22:51:35.843 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:03.646650549 +0000 2024-03-11T22:51:35.844 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:51:35.910 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:51:35.910 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:51:35.910 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000190941 s, 2.7 MB/s 2024-03-11T22:51:35.911 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:51:35.968 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/vg_nvme/lv_4 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:51:24.060194719 +0000 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:03.990642868 +0000 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:03.990642868 +0000 2024-03-11T22:51:36.023 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:03.990642868 +0000 2024-03-11T22:51:36.024 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:51:36.088 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:51:36.088 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:51:36.088 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000206564 s, 2.5 MB/s 2024-03-11T22:51:36.090 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:51:36.146 DEBUG:teuthology.orchestra.run.smithi131:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:51:36.310 INFO:teuthology.orchestra.run.smithi131.stdout:loop 2024-03-11T22:51:36.312 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_1... 2024-03-11T22:51:36.312 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:51:36.349 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-03-11T22:51:36.387 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:51:36.416 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_2... 2024-03-11T22:51:36.416 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:51:36.500 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-03-11T22:51:36.540 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:51:36.569 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_3... 2024-03-11T22:51:36.569 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:51:36.608 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-03-11T22:51:36.647 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:51:36.676 INFO:tasks.nvme_loop:Connecting nvme_loop smithi131:/dev/vg_nvme/lv_4... 2024-03-11T22:51:36.676 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:51:36.756 INFO:teuthology.orchestra.run.smithi131.stdout:1 2024-03-11T22:51:36.795 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:51:36.836 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:51:36.836 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:51:36.891 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme list 2024-03-11T22:51:36.957 INFO:teuthology.orchestra.run.smithi131.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:51:36.957 INFO:teuthology.orchestra.run.smithi131.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:51:36.957 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme0n1 CVFT623300N4400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-03-11T22:51:36.957 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme1n1 0248146d58258476bc83 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:36.957 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme2n1 f4df03a568b227a4fd1a Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:36.957 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme3n1 91a01e0503724f1d4f5c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:36.957 INFO:teuthology.orchestra.run.smithi131.stdout:/dev/nvme4n1 ccae4a2987ac0f5f25d0 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:36.958 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:51:36.958 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:51:36.959 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/scratch_devs 2024-03-11T22:51:37.025 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:51:37.025 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:51:37.044 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T22:51:37.044 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_1 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:51:24.091646931 +0000 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:02.794042891 +0000 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:02.794042891 +0000 2024-03-11T22:51:37.102 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:02.794042891 +0000 2024-03-11T22:51:37.103 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T22:51:37.172 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:51:37.172 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:51:37.172 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000120064 s, 4.3 MB/s 2024-03-11T22:51:37.173 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T22:51:37.231 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_2 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:51:24.091646931 +0000 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:03.114036942 +0000 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:03.114036942 +0000 2024-03-11T22:51:37.288 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:03.114036942 +0000 2024-03-11T22:51:37.288 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T22:51:37.357 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:51:37.357 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:51:37.357 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000105302 s, 4.9 MB/s 2024-03-11T22:51:37.358 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T22:51:37.416 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_3 2024-03-11T22:51:37.473 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T22:51:37.473 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:37.473 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-11T22:51:37.473 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:37.473 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:37.473 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:51:24.091646931 +0000 2024-03-11T22:51:37.473 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:03.441030862 +0000 2024-03-11T22:51:37.474 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:03.441030862 +0000 2024-03-11T22:51:37.474 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:03.441030862 +0000 2024-03-11T22:51:37.474 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T22:51:37.541 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:51:37.541 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:51:37.541 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 9.1055e-05 s, 5.6 MB/s 2024-03-11T22:51:37.543 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T22:51:37.599 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/vg_nvme/lv_4 2024-03-11T22:51:37.655 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:51:24.091646931 +0000 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:03.773024690 +0000 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:03.773024690 +0000 2024-03-11T22:51:37.656 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:03.773024690 +0000 2024-03-11T22:51:37.656 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T22:51:37.724 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:51:37.725 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:51:37.725 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000115084 s, 4.4 MB/s 2024-03-11T22:51:37.726 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T22:51:37.784 DEBUG:teuthology.orchestra.run.smithi195:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-03-11T22:51:37.960 INFO:teuthology.orchestra.run.smithi195.stdout:loop 2024-03-11T22:51:37.961 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_1... 2024-03-11T22:51:37.961 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-03-11T22:51:38.000 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-03-11T22:51:38.038 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_11 2024-03-11T22:51:38.066 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_2... 2024-03-11T22:51:38.067 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-03-11T22:51:38.156 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-03-11T22:51:38.195 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_21 2024-03-11T22:51:38.224 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_3... 2024-03-11T22:51:38.225 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-03-11T22:51:38.275 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-03-11T22:51:38.314 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_31 2024-03-11T22:51:38.350 INFO:tasks.nvme_loop:Connecting nvme_loop smithi195:/dev/vg_nvme/lv_4... 2024-03-11T22:51:38.350 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-03-11T22:51:38.431 INFO:teuthology.orchestra.run.smithi195.stdout:1 2024-03-11T22:51:38.474 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/vg_nvme/lv_41 2024-03-11T22:51:38.504 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:51:38.504 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:51:38.565 DEBUG:teuthology.orchestra.run.smithi195:> sudo nvme list 2024-03-11T22:51:38.631 INFO:teuthology.orchestra.run.smithi195.stdout:Node SN Model Namespace Usage Format FW Rev 2024-03-11T22:51:38.631 INFO:teuthology.orchestra.run.smithi195.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-03-11T22:51:38.631 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme0n1 PHMB751300FS480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-03-11T22:51:38.631 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme1n1 f7dc753a0085e673bb00 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:38.631 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme2n1 6e1b1dd72cad41d41a25 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:38.631 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme3n1 3a4215828f25f37a686c Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:38.631 INFO:teuthology.orchestra.run.smithi195.stdout:/dev/nvme4n1 a2c17452542ea6daf680 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-03-11T22:51:38.632 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:51:38.632 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:51:38.632 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/scratch_devs 2024-03-11T22:51:38.699 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T22:51:38.797 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '94a539707733325322964c6a187c6a1b41be1daf'} 2024-03-11T22:51:38.797 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:51:38.797 INFO:tasks.cephadm:Cluster fsid is eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:51:38.797 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T22:51:38.798 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.107', 'mon.b': '172.21.15.131', 'mon.c': '172.21.15.195'} 2024-03-11T22:51:38.798 INFO:tasks.cephadm:First mon is mon.a on smithi107 2024-03-11T22:51:38.798 INFO:tasks.cephadm:First mgr is y 2024-03-11T22:51:38.798 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T22:51:38.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-03-11T22:51:38.830 DEBUG:teuthology.orchestra.run.smithi131:> sudo hostname $(hostname -s) 2024-03-11T22:51:38.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo hostname $(hostname -s) 2024-03-11T22:51:38.890 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T22:51:38.890 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:51:39.203 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77431/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.5.35+adami05', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:08:19.449750', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '94a539707733325322964c6a187c6a1b41be1daf', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77429/', 'root_build_cause': 'SCMTRIGGER,MANUALTRIGGER', 'version': '19.0.0-1607-g94a53970', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1607.g94a53970'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 04:06:59.696700', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-11T22:51:39.583 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 94a539707733325322964c6a187c6a1b41be1daf from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:51:39.585 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm 2024-03-11T22:51:39.585 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:40.079 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:51 /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:40.080 DEBUG:teuthology.orchestra.run.smithi131:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:40.683 INFO:teuthology.orchestra.run.smithi131.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:51 /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:40.683 DEBUG:teuthology.orchestra.run.smithi195:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/94a539707733325322964c6a187c6a1b41be1daf/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:41.201 INFO:teuthology.orchestra.run.smithi195.stdout:-rw-r--r--. 1 ubuntu ubuntu 766894 Mar 11 22:51 /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:41.201 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:41.220 DEBUG:teuthology.orchestra.run.smithi131:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:41.239 DEBUG:teuthology.orchestra.run.smithi195:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T22:51:41.270 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf on all hosts... 2024-03-11T22:51:41.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:51:41.273 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:51:41.282 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf pull 2024-03-11T22:51:42.127 INFO:teuthology.orchestra.run.smithi195.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:51:42.142 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:51:42.157 INFO:teuthology.orchestra.run.smithi131.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:52:20.150 INFO:teuthology.orchestra.run.smithi131.stdout:{ 2024-03-11T22:52:20.152 INFO:teuthology.orchestra.run.smithi131.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:52:20.152 INFO:teuthology.orchestra.run.smithi131.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:52:20.152 INFO:teuthology.orchestra.run.smithi131.stdout: "repo_digests": [ 2024-03-11T22:52:20.152 INFO:teuthology.orchestra.run.smithi131.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:52:20.152 INFO:teuthology.orchestra.run.smithi131.stdout: ] 2024-03-11T22:52:20.152 INFO:teuthology.orchestra.run.smithi131.stdout:} 2024-03-11T22:53:04.505 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-03-11T22:53:04.505 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:53:04.505 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:53:04.505 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-03-11T22:53:04.505 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:53:04.505 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-03-11T22:53:04.505 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-03-11T22:53:05.237 INFO:teuthology.orchestra.run.smithi195.stdout:{ 2024-03-11T22:53:05.238 INFO:teuthology.orchestra.run.smithi195.stdout: "ceph_version": "ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev)", 2024-03-11T22:53:05.238 INFO:teuthology.orchestra.run.smithi195.stdout: "image_id": "a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e", 2024-03-11T22:53:05.238 INFO:teuthology.orchestra.run.smithi195.stdout: "repo_digests": [ 2024-03-11T22:53:05.238 INFO:teuthology.orchestra.run.smithi195.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b" 2024-03-11T22:53:05.238 INFO:teuthology.orchestra.run.smithi195.stdout: ] 2024-03-11T22:53:05.238 INFO:teuthology.orchestra.run.smithi195.stdout:} 2024-03-11T22:53:05.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-03-11T22:53:05.298 DEBUG:teuthology.orchestra.run.smithi131:> sudo mkdir -p /etc/ceph 2024-03-11T22:53:05.335 DEBUG:teuthology.orchestra.run.smithi195:> sudo mkdir -p /etc/ceph 2024-03-11T22:53:05.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-03-11T22:53:05.394 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 777 /etc/ceph 2024-03-11T22:53:05.424 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 777 /etc/ceph 2024-03-11T22:53:05.451 INFO:tasks.cephadm:Writing seed config... 2024-03-11T22:53:05.452 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T22:53:05.452 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T22:53:05.453 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T22:53:05.454 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:53:05.454 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T22:53:05.469 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = eb9eeed0-dff9-11ee-95c8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-11T22:53:05.469 DEBUG:teuthology.orchestra.run.smithi107:mon.a> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T22:53:05.511 DEBUG:teuthology.orchestra.run.smithi107:mgr.y> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T22:53:05.554 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T22:53:05.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf -v bootstrap --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.107 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:53:05.743 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-03-11T22:53:05.743 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf', '-v', 'bootstrap', '--fsid', 'eb9eeed0-dff9-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.107', '--skip-admin-label'] 2024-03-11T22:53:05.769 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:53:05.769 INFO:teuthology.orchestra.run.smithi107.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T22:53:05.769 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying podman|docker is present... 2024-03-11T22:53:05.793 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:53:05.793 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying lvm2 is present... 2024-03-11T22:53:05.793 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying time synchronization is in place... 2024-03-11T22:53:05.800 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:53:05.800 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:53:05.806 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:53:05.806 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-03-11T22:53:05.812 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-03-11T22:53:05.818 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-03-11T22:53:05.818 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:53:05.818 INFO:teuthology.orchestra.run.smithi107.stdout:Repeating the final host check... 2024-03-11T22:53:05.841 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T22:53:05.841 INFO:teuthology.orchestra.run.smithi107.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T22:53:05.841 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl is present 2024-03-11T22:53:05.841 INFO:teuthology.orchestra.run.smithi107.stdout:lvcreate is present 2024-03-11T22:53:05.847 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T22:53:05.847 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T22:53:05.852 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T22:53:05.852 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-03-11T22:53:05.858 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout enabled 2024-03-11T22:53:05.864 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout active 2024-03-11T22:53:05.864 INFO:teuthology.orchestra.run.smithi107.stdout:Unit chronyd.service is enabled and running 2024-03-11T22:53:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:Host looks OK 2024-03-11T22:53:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:Cluster fsid: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:Acquiring lock 140467619255344 on /run/cephadm/eb9eeed0-dff9-11ee-95c8-87774f69a715.lock 2024-03-11T22:53:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:Lock 140467619255344 acquired on /run/cephadm/eb9eeed0-dff9-11ee-95c8-87774f69a715.lock 2024-03-11T22:53:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 3300 ... 2024-03-11T22:53:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying IP 172.21.15.107 port 6789 ... 2024-03-11T22:53:05.865 INFO:teuthology.orchestra.run.smithi107.stdout:Base mon IP(s) is [172.21.15.107:3300, 172.21.15.107:6789], mon addrv is [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-03-11T22:53:05.868 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.107 metric 100 2024-03-11T22:53:05.868 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.107 metric 100 2024-03-11T22:53:05.871 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T22:53:05.871 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T22:53:05.871 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-03-11T22:53:05.873 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T22:53:05.873 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T22:53:05.873 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:53:05.873 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T22:53:05.873 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:47eb/64 scope link 2024-03-11T22:53:05.874 INFO:teuthology.orchestra.run.smithi107.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T22:53:05.874 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-03-11T22:53:05.874 INFO:teuthology.orchestra.run.smithi107.stdout:Mon IP `172.21.15.107` is in CIDR network `172.21.0.0/20` 2024-03-11T22:53:05.874 INFO:teuthology.orchestra.run.smithi107.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-11T22:53:05.874 INFO:teuthology.orchestra.run.smithi107.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T22:53:05.875 INFO:teuthology.orchestra.run.smithi107.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:53:06.307 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stdout a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:53:06.307 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf... 2024-03-11T22:53:06.307 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T22:53:06.307 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T22:53:06.307 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying blob sha256:0b3c7bee4e66dab4b5376aaf0baf8ba5fc1c3e66c27dc601157782d617bd9dc6 2024-03-11T22:53:06.307 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Copying config sha256:a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:53:06.307 INFO:teuthology.orchestra.run.smithi107.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T22:53:07.260 INFO:teuthology.orchestra.run.smithi107.stdout:ceph: stdout ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:53:07.260 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph version: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev) 2024-03-11T22:53:07.261 INFO:teuthology.orchestra.run.smithi107.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T22:53:08.251 INFO:teuthology.orchestra.run.smithi107.stdout:stat: stdout 167 167 2024-03-11T22:53:08.251 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial keys... 2024-03-11T22:53:09.114 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQBUi+9lY+wmIhAAUmZYRi4zWp0qBuvwtjDSUw== 2024-03-11T22:53:10.128 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQBVi+9lIJIIGhAAZZioR9Q78/HAmWZzJ6Biig== 2024-03-11T22:53:18.154 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph-authtool: stdout AQBXi+9l8fqcGBAAerjnJKmfmu5So9aPY0UR1Q== 2024-03-11T22:53:18.154 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial monmap... 2024-03-11T22:53:19.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:53:19.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T22:53:19.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:19.336 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:53:19.336 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool for a [v2:172.21.15.107:3300,v1:172.21.15.107:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T22:53:19.336 INFO:teuthology.orchestra.run.smithi107.stdout:setting min_mon_release = quincy 2024-03-11T22:53:19.336 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: set fsid to eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:19.336 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T22:53:19.336 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:53:19.336 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mon... 2024-03-11T22:53:20.323 INFO:teuthology.orchestra.run.smithi107.stdout:create mon.a on 2024-03-11T22:53:20.675 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T22:53:20.847 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715.target. 2024-03-11T22:53:20.847 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715.target. 2024-03-11T22:53:21.073 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:53:21.074 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T22:53:21.237 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715.target.wants/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@.service. 2024-03-11T22:53:21.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:21 smithi107 podman[25321]: 2024-03-11 22:53:21.469385164 +0000 UTC m=+0.115216252 container create 3356e9175028763ac07df6cfdd21acb1149a2ae29fec5f5fad6e39330491dff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397) 2024-03-11T22:53:21.780 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:53:21.780 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-03-11T22:53:21.787 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:53:21.787 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-03-11T22:53:21.787 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-03-11T22:53:21.787 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:53:21.787 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon to start... 2024-03-11T22:53:21.787 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mon... 2024-03-11T22:53:21.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:21 smithi107 podman[25321]: 2024-03-11 22:53:21.671645088 +0000 UTC m=+0.317476134 container init 3356e9175028763ac07df6cfdd21acb1149a2ae29fec5f5fad6e39330491dff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-11T22:53:21.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:21 smithi107 podman[25321]: 2024-03-11 22:53:21.676703889 +0000 UTC m=+0.322534936 container start 3356e9175028763ac07df6cfdd21acb1149a2ae29fec5f5fad6e39330491dff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T22:53:21.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:21 smithi107 ceph-mon[25359]: mkfs eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:21.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:21 smithi107 ceph-mon[25359]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:53:21.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:21 smithi107 bash[25321]: 3356e9175028763ac07df6cfdd21acb1149a2ae29fec5f5fad6e39330491dff6 2024-03-11T22:53:21.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:21 smithi107 systemd[1]: Started Ceph mon.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:22.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:22 smithi107 ceph-mon[25359]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:53:22.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:22 smithi107 ceph-mon[25359]: monmap e1: 1 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:53:22.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:22 smithi107 ceph-mon[25359]: fsmap 2024-03-11T22:53:22.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:22 smithi107 ceph-mon[25359]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:53:22.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:22 smithi107 ceph-mon[25359]: mgrmap e1: no daemons active 2024-03-11T22:53:22.866 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:22 smithi107 ceph-mon[25359]: from='client.? 172.21.15.107:0/1732613187' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout id: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout services: 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.546252s) 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout data: 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T22:53:22.870 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T22:53:22.871 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T22:53:22.871 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T22:53:22.871 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:22.871 INFO:teuthology.orchestra.run.smithi107.stdout:mon is available 2024-03-11T22:53:22.871 INFO:teuthology.orchestra.run.smithi107.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T22:53:23.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:23 smithi107 ceph-mon[25359]: from='client.? 172.21.15.107:0/2633762595' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:53:23.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:23 smithi107 ceph-mon[25359]: from='client.? 172.21.15.107:0/2633762595' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:53:25.511 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:25.511 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:53:25.511 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:25.511 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.107:3300,v1:172.21.15.107:6789] 2024-03-11T22:53:25.511 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:53:25.511 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:53:25.512 INFO:teuthology.orchestra.run.smithi107.stdout:Generating new minimal ceph.conf... 2024-03-11T22:53:26.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:26 smithi107 ceph-mon[25359]: from='client.? 172.21.15.107:0/3398908397' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:53:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Restarting the monitor... 2024-03-11T22:53:26.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:26 smithi107 systemd[1]: Stopping Ceph mon.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:53:27.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:26 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a[25336]: 2024-03-11T22:53:26.838+0000 7fd117361700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:53:27.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:26 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a[25336]: 2024-03-11T22:53:26.838+0000 7fd117361700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T22:53:27.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:27 smithi107 podman[25692]: 2024-03-11 22:53:27.023870842 +0000 UTC m=+0.325241096 container died 3356e9175028763ac07df6cfdd21acb1149a2ae29fec5f5fad6e39330491dff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307) 2024-03-11T22:53:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:27 smithi107 podman[25692]: 2024-03-11 22:53:27.377627843 +0000 UTC m=+0.678998095 container cleanup 3356e9175028763ac07df6cfdd21acb1149a2ae29fec5f5fad6e39330491dff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T22:53:27.666 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:27 smithi107 bash[25692]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:53:27.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:27 smithi107 podman[25711]: 2024-03-11 22:53:27.661778097 +0000 UTC m=+0.640865327 container remove 3356e9175028763ac07df6cfdd21acb1149a2ae29fec5f5fad6e39330491dff6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-11T22:53:27.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:27 smithi107 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T22:53:27.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:27 smithi107 systemd[1]: Stopped Ceph mon.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:27.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:27 smithi107 systemd[1]: Starting Ceph mon.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 podman[25811]: 2024-03-11 22:53:27.911294598 +0000 UTC m=+0.017596151 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 podman[25811]: 2024-03-11 22:53:28.012737064 +0000 UTC m=+0.119038655 container create 942d73530b9a87182bae861e4d2296177a95705c40e63f48139c1fd732e57460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 podman[25811]: 2024-03-11 22:53:28.198266086 +0000 UTC m=+0.304567646 container init 942d73530b9a87182bae861e4d2296177a95705c40e63f48139c1fd732e57460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 podman[25811]: 2024-03-11 22:53:28.203799486 +0000 UTC m=+0.310101049 container start 942d73530b9a87182bae861e4d2296177a95705c40e63f48139c1fd732e57460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: pidfile_write: ignore empty --pid-file 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: load: jerasure load: lrc 2024-03-11T22:53:28.286 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Git sha 0 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: DB SUMMARY 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: DB Session ID: 9SIQ13ZDQXW2USWF2F5N 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 84657 ; 2024-03-11T22:53:28.287 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:53:28.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:53:28.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:53:28.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:53:28.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:53:28.288 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.env: 0x559dc9924ae0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.info_log: 0x559dcb6438c0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.statistics: (nil) 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.use_fsync: 0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:53:28.289 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.db_log_dir: 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.wal_dir: 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.write_buffer_manager: 0x559dcb617400 2024-03-11T22:53:28.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.unordered_write: 0 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.row_cache: None 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.wal_filter: None 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:53:28.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.wal_compression: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:53:28.292 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_open_files: -1 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:53:28.293 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Compression algorithms supported: 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kZSTD supported: 0 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:53:28.294 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.merge_operator: 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_filter: None 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559dcb6439c0) 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_type: 0 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_index_type: 0 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_shortening: 1 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:53:28.295 INFO:journalctl@ceph.mon.a.smithi107.stdout: checksum: 4 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: no_block_cache: 0 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache: 0x559dcb647090 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_options: 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: capacity : 536870912 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_shard_bits : 4 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: strict_capacity_limit : 0 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_cache_compressed: (nil) 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: persistent_cache: (nil) 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size: 4096 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_size_deviation: 10 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_restart_interval: 16 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: index_block_restart_interval: 1 2024-03-11T22:53:28.296 INFO:journalctl@ceph.mon.a.smithi107.stdout: metadata_block_size: 4096 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: partition_filters: 0 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: use_delta_encoding: 1 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: filter_policy: bloomfilter 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: whole_key_filtering: 1 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: verify_compression: 0 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: format_version: 5 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_index_compression: 1 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: block_align: 0 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: max_auto_readahead_size: 262144 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: prepopulate_block_cache: 0 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression: NoCompression 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:53:28.297 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.num_levels: 7 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:53:28.298 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:53:28.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:53:28.300 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:53:28.301 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:53:28.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:53:28.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:53:28.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:53:28.302 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.ttl: 2592000 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:53:28.303 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a9caa6e1-4f83-4f01-8f0b-a035bd444178 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197608232443, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197608233285, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81282, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 79469, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 73778, "raw_average_value_size": 349, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710197608, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a9caa6e1-4f83-4f01-8f0b-a035bd444178", "db_session_id": "9SIQ13ZDQXW2USWF2F5N", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197608233366, "job": 1, "event": "recovery_finished"} 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559dcb738000 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: DB pointer 0x559dcb722000 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:53:28.304 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** DB Stats ** 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: L0 2/0 81.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Sum 2/0 81.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:53:28.305 INFO:journalctl@ceph.mon.a.smithi107.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** Compaction Stats [default] ** 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.6 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:53:28.306 INFO:journalctl@ceph.mon.a.smithi107.stdout: Cumulative compaction: 0.00 GB write, 27.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: Interval compaction: 0.00 GB write, 27.93 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache BinnedLRUCache@0x559dcb647090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: starting mon.a rank 0 at public addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] at bind addrs [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???) e1 preinit fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???).mds e1 new map 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???).mds e1 print_map 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: e1 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: legacy client fscid: -1 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: 2024-03-11T22:53:28.307 INFO:journalctl@ceph.mon.a.smithi107.stdout: No filesystems configured 2024-03-11T22:53:28.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:53:28.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:53:28.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:53:28.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:53:28.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T22:53:28.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 bash[25811]: 942d73530b9a87182bae861e4d2296177a95705c40e63f48139c1fd732e57460 2024-03-11T22:53:28.308 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 systemd[1]: Started Ceph mon.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:28.308 INFO:teuthology.orchestra.run.smithi107.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T22:53:28.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T22:53:28.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: monmap e1: 1 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:53:28.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: fsmap 2024-03-11T22:53:28.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T22:53:28.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:28 smithi107 ceph-mon[25849]: mgrmap e1: no daemons active 2024-03-11T22:53:29.454 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T22:53:29.475 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:53:29.475 INFO:teuthology.orchestra.run.smithi107.stdout:Creating mgr... 2024-03-11T22:53:29.476 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T22:53:29.476 INFO:teuthology.orchestra.run.smithi107.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T22:53:29.715 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:53:29.715 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Failed to reset failed state of unit ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T22:53:29.866 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715.target.wants/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@.service. 2024-03-11T22:53:30.094 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:29 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/4278232626' entity='client.admin' 2024-03-11T22:53:30.095 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:29 smithi107 systemd[1]: Starting Ceph mgr.y for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:53:30.342 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:53:30.343 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-03-11T22:53:30.349 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:53:30.350 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-03-11T22:53:30.350 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-03-11T22:53:30.350 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T22:53:30.358 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:53:30.358 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-03-11T22:53:30.365 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:53:30.365 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-03-11T22:53:30.365 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-03-11T22:53:30.366 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T22:53:30.366 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr to start... 2024-03-11T22:53:30.366 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr... 2024-03-11T22:53:30.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 podman[26091]: 2024-03-11 22:53:30.089976245 +0000 UTC m=+0.109353527 container create 234ee877afdfc8c791ac2f75d2d84fc7113e9abe4ed6108839aa3d1d1b8a61d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-94a5397) 2024-03-11T22:53:30.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 podman[26091]: 2024-03-11 22:53:29.99815313 +0000 UTC m=+0.017530421 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf 2024-03-11T22:53:30.453 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 podman[26091]: 2024-03-11 22:53:30.275694729 +0000 UTC m=+0.295072017 container init 234ee877afdfc8c791ac2f75d2d84fc7113e9abe4ed6108839aa3d1d1b8a61d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T22:53:30.453 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 podman[26091]: 2024-03-11 22:53:30.28065664 +0000 UTC m=+0.300033923 container start 234ee877afdfc8c791ac2f75d2d84fc7113e9abe4ed6108839aa3d1d1b8a61d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T22:53:30.453 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 bash[26091]: 234ee877afdfc8c791ac2f75d2d84fc7113e9abe4ed6108839aa3d1d1b8a61d0 2024-03-11T22:53:30.453 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 systemd[1]: Started Ceph mgr.y for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:53:31.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:30 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/2602364225' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:53:31.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:30.835+0000 7f2d24a60200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:53:31.203 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:30 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:30.937+0000 7f2d24a60200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:53:31.699 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:31.699 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:31.699 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "eb9eeed0-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:53:31.700 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:53:31.701 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:53:31.704 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:53:31.704 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:53:31.705 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:53:21.743710+0000", 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:31.706 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:31.707 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:53:31.707 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:31.707 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (1/15)... 2024-03-11T22:53:31.821 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:31 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:31.581+0000 7f2d24a60200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:53:31.821 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:31 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:31.674+0000 7f2d24a60200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:53:31.821 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:31 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:31.816+0000 7f2d24a60200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:53:32.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:31 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:31.911+0000 7f2d24a60200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:53:32.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:31 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:31.993+0000 7f2d24a60200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:53:32.541 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:32.252+0000 7f2d24a60200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:53:32.541 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:32.359+0000 7f2d24a60200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:53:32.541 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:32.443+0000 7f2d24a60200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:53:32.541 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:32.536+0000 7f2d24a60200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:53:32.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:32.709+0000 7f2d24a60200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:53:33.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:33 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:33.655+0000 7f2d24a60200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:53:33.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:33 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:33.790+0000 7f2d24a60200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:53:34.237 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:34.138+0000 7f2d24a60200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:53:34.238 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:34.232+0000 7f2d24a60200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:53:34.626 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:34 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/278243053' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:53:34.626 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:34.437+0000 7f2d24a60200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:53:34.884 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:34.628+0000 7f2d24a60200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "eb9eeed0-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:34.893 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:53:34.894 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:53:34.895 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:34.896 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:53:21.743710+0000", 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:34.897 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (2/15)... 2024-03-11T22:53:35.179 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:34.919+0000 7f2d24a60200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:53:35.179 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:35 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:35.174+0000 7f2d24a60200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:53:35.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:35 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:35.405+0000 7f2d24a60200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:53:36.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:36 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:36.187+0000 7f2d24a60200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:53:37.015 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:36 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:36.744+0000 7f2d24a60200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:53:37.016 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:36 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:36.841+0000 7f2d24a60200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:37.055+0000 7f2d24a60200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: Activating manager daemon y 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: mgrmap e2: y(active, starting, since 0.00310136s) 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:53:37.354 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:53:37.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: Manager daemon y is now available 2024-03-11T22:53:37.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:53:37.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' 2024-03-11T22:53:37.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:53:37.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' 2024-03-11T22:53:37.355 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:37 smithi107 ceph-mon[25849]: from='mgr.14100 172.21.15.107:0/4011306388' entity='mgr.y' 2024-03-11T22:53:37.987 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "eb9eeed0-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:53:37.988 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:53:37.989 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:53:37.993 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:53:37.994 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:53:21.743710+0000", 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:53:37.995 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:37.996 INFO:teuthology.orchestra.run.smithi107.stdout:mgr not available, waiting (3/15)... 2024-03-11T22:53:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:38 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/118157366' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:53:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:38 smithi107 ceph-mon[25849]: mgrmap e3: y(active, since 1.00572s) 2024-03-11T22:53:40.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:40 smithi107 ceph-mon[25849]: mgrmap e4: y(active, since 2s) 2024-03-11T22:53:41.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:41 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/4217189022' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T22:53:41.326 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsid": "eb9eeed0-dff9-11ee-95c8-87774f69a715", 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 0 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T22:53:41.327 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T22:53:41.328 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T22:53:41.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T22:53:41.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T22:53:41.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T22:53:41.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:41.332 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:41.333 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ], 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T22:53:21.743710+0000", 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout }, 2024-03-11T22:53:41.334 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T22:53:41.335 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:41.335 INFO:teuthology.orchestra.run.smithi107.stdout:mgr is available 2024-03-11T22:53:42.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:42 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/810553339' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T22:53:42.319 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:42 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/810553339' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-11T22:53:42.581 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout fsid = eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T22:53:42.582 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling cephadm module... 2024-03-11T22:53:43.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:43 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3699992509' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T22:53:43.659 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ignoring --setuser ceph since I am not root 2024-03-11T22:53:43.659 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ignoring --setgroup ceph since I am not root 2024-03-11T22:53:43.934 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:43.892+0000 7f5f9f51e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:53:44.191 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:43.989+0000 7f5f9f51e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:53:44.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:44 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3699992509' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T22:53:44.448 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:44 smithi107 ceph-mon[25849]: mgrmap e5: y(active, since 6s) 2024-03-11T22:53:44.702 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:44 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:44.669+0000 7f5f9f51e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:53:45.014 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:44 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:44.774+0000 7f5f9f51e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:53:45.015 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:44 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:44.913+0000 7f5f9f51e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:53:45.015 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:45.009+0000 7f5f9f51e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:53:45.284 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:45.092+0000 7f5f9f51e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:53:45.287 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:45.287 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T22:53:45.287 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:53:45.288 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:53:45.288 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:53:45.288 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:45.288 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-03-11T22:53:45.288 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 5... 2024-03-11T22:53:45.564 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3270544536' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:53:45.564 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:45.358+0000 7f5f9f51e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:53:45.564 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:45.468+0000 7f5f9f51e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:53:45.564 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:45.559+0000 7f5f9f51e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:53:45.837 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:45.655+0000 7f5f9f51e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:53:46.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:45 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:45.831+0000 7f5f9f51e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:53:47.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:46 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:46.769+0000 7f5f9f51e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:53:47.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:46 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:46.897+0000 7f5f9f51e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:53:47.516 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:47 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:47.225+0000 7f5f9f51e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:53:47.516 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:47 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:47.316+0000 7f5f9f51e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:53:47.517 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:47 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:47.511+0000 7f5f9f51e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:53:47.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:47 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:47.696+0000 7f5f9f51e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:53:48.264 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:47 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:47.980+0000 7f5f9f51e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:53:48.264 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:48 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:48.259+0000 7f5f9f51e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:53:48.702 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:48 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:48.495+0000 7f5f9f51e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:53:49.702 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:49 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:49.295+0000 7f5f9f51e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:53:50.190 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:49 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:49.874+0000 7f5f9f51e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:53:50.190 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:49 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:49.971+0000 7f5f9f51e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:53:50.190 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:53:50.185+0000 7f5f9f51e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:53:50.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: Active manager daemon y restarted 2024-03-11T22:53:50.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: Activating manager daemon y 2024-03-11T22:53:50.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T22:53:50.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: mgrmap e6: y(active, starting, since 0.00323441s) 2024-03-11T22:53:50.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:53:50.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:53:50.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: Manager daemon y is now available 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:50.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:50 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:51.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:51 smithi107 ceph-mon[25849]: Found migration_current of "None". Setting to last migration. 2024-03-11T22:53:51.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:51 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:51.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:51 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:53:51.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:51 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:53:51.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:51 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:51.565 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:51 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:51.566 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:51 smithi107 ceph-mon[25849]: mgrmap e7: y(active, since 1.00553s) 2024-03-11T22:53:51.810 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:53:51.810 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T22:53:51.810 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:53:51.810 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:53:51.810 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 5 is available 2024-03-11T22:53:51.810 INFO:teuthology.orchestra.run.smithi107.stdout:Setting orchestrator backend to cephadm... 2024-03-11T22:53:52.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:52 smithi107 ceph-mon[25849]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:53:52.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:52 smithi107 ceph-mon[25849]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:53:52.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:52 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:53.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:53 smithi107 ceph-mon[25849]: [11/Mar/2024:22:53:51] ENGINE Bus STARTING 2024-03-11T22:53:53.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:53 smithi107 ceph-mon[25849]: [11/Mar/2024:22:53:51] ENGINE Serving on http://172.21.15.107:8765 2024-03-11T22:53:53.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:53 smithi107 ceph-mon[25849]: [11/Mar/2024:22:53:51] ENGINE Serving on https://172.21.15.107:7150 2024-03-11T22:53:53.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:53 smithi107 ceph-mon[25849]: [11/Mar/2024:22:53:51] ENGINE Bus STARTED 2024-03-11T22:53:53.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:53 smithi107 ceph-mon[25849]: mgrmap e8: y(active, since 2s) 2024-03-11T22:53:53.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:53 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:53.426 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:53 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:53:54.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:54 smithi107 ceph-mon[25849]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:54.372 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T22:53:54.372 INFO:teuthology.orchestra.run.smithi107.stdout:Generating ssh key... 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-mon[25849]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: Generating public/private rsa key pair. 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: Your identification has been saved in /tmp/tmp4fu6oi6b/key. 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: Your public key has been saved in /tmp/tmp4fu6oi6b/key.pub. 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: The key fingerprint is: 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: SHA256:V/GHNHHPvmIvF2TeA+SYFeYnOkBPBI140n2dUODW8T0 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: The key's randomart image is: 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: +---[RSA 3072]----+ 2024-03-11T22:53:55.685 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | oo*o.=X+o| 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | o.+oo+B.B=| 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | o. .O=oE*| 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | .+ooo=.| 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | S .o = o| 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | . . +o| 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | o .o| 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | ..o. | 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: | o. | 2024-03-11T22:53:55.686 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:53:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: +----[SHA256]-----+ 2024-03-11T22:53:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:56 smithi107 ceph-mon[25849]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:56 smithi107 ceph-mon[25849]: Generating ssh key... 2024-03-11T22:53:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:56 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:56 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:53:57.185 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE+kIPBzl3rouHJs/IysrdCTa3DdR3cKDzVZAWTuvlq0kSi6fBtzGXD+iQo+WPjtTQw0ZUHdgjHFQjAFPnAXW1JAchh2U5VHMMuGf3e+cCPn+Vpj2ZNNJFSkic4xA26WEz7ftg2jkZ1uaqsur2Hycjw5fmRbSKVYQZU+FgZq7U3J8MSfzu/EYZb2AEwT4hscnuE5kYgOizhpJNFeLAtE71Mg9szVVAvwaO4O0zf5F72TN5a4n2EHpMvrKvD8dhPct4ZRNwmuN9kh86ovTwg3jZgQxpzor65y7yz0P+w+TBfxA2H8/xZwZwR5UsQY1Ad0u2TcYGJU+RDvL9QZ5jKxKpuxKrZeh0duayGkDMg6dfNvvO1lC+i8z6SH6hut2rDwRvmGZcveeWhSnDMi7VJFT/WgVe7CRJbx/OxSNvOvlVViTDI+47c0BeZkXR+QC/wRBsZNRLc4+X07FNpha/XtYSEwaCUO6wIyiN6T55JOkFmNqp9fBEHWG04LfY1N2/IYM= ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:53:57.185 INFO:teuthology.orchestra.run.smithi107.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:53:57.185 INFO:teuthology.orchestra.run.smithi107.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T22:53:57.185 INFO:teuthology.orchestra.run.smithi107.stdout:Adding host smithi107... 2024-03-11T22:53:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:57 smithi107 ceph-mon[25849]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:58.538 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:58 smithi107 ceph-mon[25849]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "addr": "172.21.15.107", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:53:59.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:53:59 smithi107 ceph-mon[25849]: Deploying cephadm binary to smithi107 2024-03-11T22:54:02.684 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Added host 'smithi107' with addr '172.21.15.107' 2024-03-11T22:54:02.685 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mon service... 2024-03-11T22:54:03.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:03 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:03.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:03 smithi107 ceph-mon[25849]: Added host smithi107 2024-03-11T22:54:03.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:03 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:03.984 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T22:54:03.984 INFO:teuthology.orchestra.run.smithi107.stdout:Deploying unmanaged mgr service... 2024-03-11T22:54:04.586 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:04 smithi107 ceph-mon[25849]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:04.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:04 smithi107 ceph-mon[25849]: Saving service mon spec with placement count:5 2024-03-11T22:54:04.587 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:04 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:05.552 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T22:54:05.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:05 smithi107 ceph-mon[25849]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:05.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:05 smithi107 ceph-mon[25849]: Saving service mgr spec with placement count:2 2024-03-11T22:54:05.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:05 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:07.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:07 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1585330257' entity='client.admin' 2024-03-11T22:54:07.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:07 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:07.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:07 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:08.537 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling the dashboard module... 2024-03-11T22:54:09.113 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:08 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/572606875' entity='client.admin' 2024-03-11T22:54:10.619 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ignoring --setuser ceph since I am not root 2024-03-11T22:54:10.620 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ignoring --setgroup ceph since I am not root 2024-03-11T22:54:10.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:10.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:10.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:54:10.620 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' 2024-03-11T22:54:10.621 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi107", "caps": []}]: dispatch 2024-03-11T22:54:10.621 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-mon[25849]: from='mgr.14120 172.21.15.107:0/3390655727' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi107", "caps": []}]': finished 2024-03-11T22:54:10.621 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3575566531' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T22:54:10.892 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:10.847+0000 7f2eeb015200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:54:11.155 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:10 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:10.944+0000 7f2eeb015200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:54:11.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:11 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3575566531' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T22:54:11.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:11 smithi107 ceph-mon[25849]: mgrmap e9: y(active, since 20s) 2024-03-11T22:54:11.924 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:11 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:11.593+0000 7f2eeb015200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:54:11.924 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:11 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:11.689+0000 7f2eeb015200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:54:11.924 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:11 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:11.831+0000 7f2eeb015200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:54:12.199 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:11 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:11.927+0000 7f2eeb015200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:54:12.199 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:12 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:12.009+0000 7f2eeb015200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:54:12.201 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:54:12.202 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T22:54:12.202 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T22:54:12.202 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T22:54:12.202 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T22:54:12.202 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:54:12.202 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for the mgr to restart... 2024-03-11T22:54:12.202 INFO:teuthology.orchestra.run.smithi107.stdout:Waiting for mgr epoch 9... 2024-03-11T22:54:12.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:12 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/453480546' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T22:54:12.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:12 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:12.276+0000 7f2eeb015200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:54:12.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:12 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:12.386+0000 7f2eeb015200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:54:12.768 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:12 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:12.486+0000 7f2eeb015200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:54:12.768 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:12 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:12.582+0000 7f2eeb015200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:54:13.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:12 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:12.763+0000 7f2eeb015200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:54:14.189 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:13 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:13.721+0000 7f2eeb015200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:54:14.190 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:13 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:13.852+0000 7f2eeb015200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:54:14.190 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:14 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:14.184+0000 7f2eeb015200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:54:14.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:14 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:14.274+0000 7f2eeb015200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:54:14.873 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:14 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:14.469+0000 7f2eeb015200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:54:14.873 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:14 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:14.654+0000 7f2eeb015200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:54:15.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:14 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:14.942+0000 7f2eeb015200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:54:15.468 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:15 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:15.199+0000 7f2eeb015200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:54:15.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:15 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:15.467+0000 7f2eeb015200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:54:16.702 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:16 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:16.268+0000 7f2eeb015200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:54:17.189 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:16 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:16.864+0000 7f2eeb015200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:54:17.189 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:16 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:16.964+0000 7f2eeb015200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:54:17.189 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:54:17.184+0000 7f2eeb015200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:54:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: Active manager daemon y restarted 2024-03-11T22:54:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: Activating manager daemon y 2024-03-11T22:54:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T22:54:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: mgrmap e10: y(active, starting, since 0.00319224s) 2024-03-11T22:54:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:54:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:54:17.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:54:17.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:54:17.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:54:17.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: Manager daemon y is now available 2024-03-11T22:54:17.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:17.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:17.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:18.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:54:18.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:54:18.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:18 smithi107 ceph-mon[25849]: mgrmap e11: y(active, since 1.00533s) 2024-03-11T22:54:19.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout { 2024-03-11T22:54:19.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T22:54:19.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T22:54:19.136 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout } 2024-03-11T22:54:19.136 INFO:teuthology.orchestra.run.smithi107.stdout:mgr epoch 9 is available 2024-03-11T22:54:19.136 INFO:teuthology.orchestra.run.smithi107.stdout:Generating a dashboard self-signed certificate... 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: [11/Mar/2024:22:54:18] ENGINE Bus STARTING 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: [11/Mar/2024:22:54:18] ENGINE Serving on https://172.21.15.107:7150 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: [11/Mar/2024:22:54:18] ENGINE Serving on http://172.21.15.107:8765 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: [11/Mar/2024:22:54:18] ENGINE Bus STARTED 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:54:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:19.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:19 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi107", "caps": []}]: dispatch 2024-03-11T22:54:20.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:20 smithi107 ceph-mon[25849]: Deploying daemon agent.smithi107 on smithi107 2024-03-11T22:54:20.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:20 smithi107 ceph-mon[25849]: mgrmap e12: y(active, since 2s) 2024-03-11T22:54:20.322 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:20 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:20.323 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:20 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:20.583 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T22:54:20.583 INFO:teuthology.orchestra.run.smithi107.stdout:Creating initial admin user... 2024-03-11T22:54:21.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:21 smithi107 ceph-mon[25849]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:22.057 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$3f8TazOWgG6B9q7jX8NKEuZDXJER5A.nCEjbqTzsNOGk4Dot3H3Lm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710197661, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T22:54:22.057 INFO:teuthology.orchestra.run.smithi107.stdout:Fetching dashboard port number... 2024-03-11T22:54:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:22 smithi107 ceph-mon[25849]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:22.432 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:22 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:23.336 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:23 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/950599374' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T22:54:23.338 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T22:54:23.347 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T22:54:23.347 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout disabled 2024-03-11T22:54:23.354 INFO:teuthology.orchestra.run.smithi107.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T22:54:23.354 INFO:teuthology.orchestra.run.smithi107.stdout:systemctl: stdout inactive 2024-03-11T22:54:23.354 INFO:teuthology.orchestra.run.smithi107.stdout:firewalld.service is not enabled 2024-03-11T22:54:23.354 INFO:teuthology.orchestra.run.smithi107.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T22:54:23.356 INFO:teuthology.orchestra.run.smithi107.stdout:Ceph Dashboard is now available at: 2024-03-11T22:54:23.356 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: URL: https://smithi107.front.sepia.ceph.com:8443/ 2024-03-11T22:54:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: User: admin 2024-03-11T22:54:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: Password: krz6ycbqyd 2024-03-11T22:54:23.357 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:23.357 INFO:teuthology.orchestra.run.smithi107.stdout:Saving cluster configuration to /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config directory 2024-03-11T22:54:23.357 INFO:teuthology.orchestra.run.smithi107.stdout:Enabling autotune for osd_memory_target 2024-03-11T22:54:24.533 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:24 smithi107 ceph-mon[25849]: mgrmap e13: y(active, since 6s) 2024-03-11T22:54:25.875 INFO:teuthology.orchestra.run.smithi107.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T22:54:25.875 INFO:teuthology.orchestra.run.smithi107.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T22:54:25.875 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: ceph telemetry on 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout:For more information see: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:25.876 INFO:teuthology.orchestra.run.smithi107.stdout:Bootstrap complete. 2024-03-11T22:54:25.914 INFO:tasks.cephadm:Fetching config... 2024-03-11T22:54:25.914 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:54:25.914 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T22:54:25.929 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T22:54:25.929 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:54:25.929 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T22:54:25.985 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T22:54:25.985 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:54:25.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T22:54:26.147 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T22:54:26.147 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:54:26.147 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T22:54:26.208 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T22:54:26.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE+kIPBzl3rouHJs/IysrdCTa3DdR3cKDzVZAWTuvlq0kSi6fBtzGXD+iQo+WPjtTQw0ZUHdgjHFQjAFPnAXW1JAchh2U5VHMMuGf3e+cCPn+Vpj2ZNNJFSkic4xA26WEz7ftg2jkZ1uaqsur2Hycjw5fmRbSKVYQZU+FgZq7U3J8MSfzu/EYZb2AEwT4hscnuE5kYgOizhpJNFeLAtE71Mg9szVVAvwaO4O0zf5F72TN5a4n2EHpMvrKvD8dhPct4ZRNwmuN9kh86ovTwg3jZgQxpzor65y7yz0P+w+TBfxA2H8/xZwZwR5UsQY1Ad0u2TcYGJU+RDvL9QZ5jKxKpuxKrZeh0duayGkDMg6dfNvvO1lC+i8z6SH6hut2rDwRvmGZcveeWhSnDMi7VJFT/WgVe7CRJbx/OxSNvOvlVViTDI+47c0BeZkXR+QC/wRBsZNRLc4+X07FNpha/XtYSEwaCUO6wIyiN6T55JOkFmNqp9fBEHWG04LfY1N2/IYM= ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:54:26.349 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE+kIPBzl3rouHJs/IysrdCTa3DdR3cKDzVZAWTuvlq0kSi6fBtzGXD+iQo+WPjtTQw0ZUHdgjHFQjAFPnAXW1JAchh2U5VHMMuGf3e+cCPn+Vpj2ZNNJFSkic4xA26WEz7ftg2jkZ1uaqsur2Hycjw5fmRbSKVYQZU+FgZq7U3J8MSfzu/EYZb2AEwT4hscnuE5kYgOizhpJNFeLAtE71Mg9szVVAvwaO4O0zf5F72TN5a4n2EHpMvrKvD8dhPct4ZRNwmuN9kh86ovTwg3jZgQxpzor65y7yz0P+w+TBfxA2H8/xZwZwR5UsQY1Ad0u2TcYGJU+RDvL9QZ5jKxKpuxKrZeh0duayGkDMg6dfNvvO1lC+i8z6SH6hut2rDwRvmGZcveeWhSnDMi7VJFT/WgVe7CRJbx/OxSNvOvlVViTDI+47c0BeZkXR+QC/wRBsZNRLc4+X07FNpha/XtYSEwaCUO6wIyiN6T55JOkFmNqp9fBEHWG04LfY1N2/IYM= ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:54:26.363 DEBUG:teuthology.orchestra.run.smithi131:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE+kIPBzl3rouHJs/IysrdCTa3DdR3cKDzVZAWTuvlq0kSi6fBtzGXD+iQo+WPjtTQw0ZUHdgjHFQjAFPnAXW1JAchh2U5VHMMuGf3e+cCPn+Vpj2ZNNJFSkic4xA26WEz7ftg2jkZ1uaqsur2Hycjw5fmRbSKVYQZU+FgZq7U3J8MSfzu/EYZb2AEwT4hscnuE5kYgOizhpJNFeLAtE71Mg9szVVAvwaO4O0zf5F72TN5a4n2EHpMvrKvD8dhPct4ZRNwmuN9kh86ovTwg3jZgQxpzor65y7yz0P+w+TBfxA2H8/xZwZwR5UsQY1Ad0u2TcYGJU+RDvL9QZ5jKxKpuxKrZeh0duayGkDMg6dfNvvO1lC+i8z6SH6hut2rDwRvmGZcveeWhSnDMi7VJFT/WgVe7CRJbx/OxSNvOvlVViTDI+47c0BeZkXR+QC/wRBsZNRLc4+X07FNpha/XtYSEwaCUO6wIyiN6T55JOkFmNqp9fBEHWG04LfY1N2/IYM= ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:54:26.408 INFO:teuthology.orchestra.run.smithi131.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE+kIPBzl3rouHJs/IysrdCTa3DdR3cKDzVZAWTuvlq0kSi6fBtzGXD+iQo+WPjtTQw0ZUHdgjHFQjAFPnAXW1JAchh2U5VHMMuGf3e+cCPn+Vpj2ZNNJFSkic4xA26WEz7ftg2jkZ1uaqsur2Hycjw5fmRbSKVYQZU+FgZq7U3J8MSfzu/EYZb2AEwT4hscnuE5kYgOizhpJNFeLAtE71Mg9szVVAvwaO4O0zf5F72TN5a4n2EHpMvrKvD8dhPct4ZRNwmuN9kh86ovTwg3jZgQxpzor65y7yz0P+w+TBfxA2H8/xZwZwR5UsQY1Ad0u2TcYGJU+RDvL9QZ5jKxKpuxKrZeh0duayGkDMg6dfNvvO1lC+i8z6SH6hut2rDwRvmGZcveeWhSnDMi7VJFT/WgVe7CRJbx/OxSNvOvlVViTDI+47c0BeZkXR+QC/wRBsZNRLc4+X07FNpha/XtYSEwaCUO6wIyiN6T55JOkFmNqp9fBEHWG04LfY1N2/IYM= ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:54:26.412 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:26 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/2305916088' entity='client.admin' 2024-03-11T22:54:26.423 DEBUG:teuthology.orchestra.run.smithi195:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE+kIPBzl3rouHJs/IysrdCTa3DdR3cKDzVZAWTuvlq0kSi6fBtzGXD+iQo+WPjtTQw0ZUHdgjHFQjAFPnAXW1JAchh2U5VHMMuGf3e+cCPn+Vpj2ZNNJFSkic4xA26WEz7ftg2jkZ1uaqsur2Hycjw5fmRbSKVYQZU+FgZq7U3J8MSfzu/EYZb2AEwT4hscnuE5kYgOizhpJNFeLAtE71Mg9szVVAvwaO4O0zf5F72TN5a4n2EHpMvrKvD8dhPct4ZRNwmuN9kh86ovTwg3jZgQxpzor65y7yz0P+w+TBfxA2H8/xZwZwR5UsQY1Ad0u2TcYGJU+RDvL9QZ5jKxKpuxKrZeh0duayGkDMg6dfNvvO1lC+i8z6SH6hut2rDwRvmGZcveeWhSnDMi7VJFT/WgVe7CRJbx/OxSNvOvlVViTDI+47c0BeZkXR+QC/wRBsZNRLc4+X07FNpha/XtYSEwaCUO6wIyiN6T55JOkFmNqp9fBEHWG04LfY1N2/IYM= ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T22:54:26.468 INFO:teuthology.orchestra.run.smithi195.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDE+kIPBzl3rouHJs/IysrdCTa3DdR3cKDzVZAWTuvlq0kSi6fBtzGXD+iQo+WPjtTQw0ZUHdgjHFQjAFPnAXW1JAchh2U5VHMMuGf3e+cCPn+Vpj2ZNNJFSkic4xA26WEz7ftg2jkZ1uaqsur2Hycjw5fmRbSKVYQZU+FgZq7U3J8MSfzu/EYZb2AEwT4hscnuE5kYgOizhpJNFeLAtE71Mg9szVVAvwaO4O0zf5F72TN5a4n2EHpMvrKvD8dhPct4ZRNwmuN9kh86ovTwg3jZgQxpzor65y7yz0P+w+TBfxA2H8/xZwZwR5UsQY1Ad0u2TcYGJU+RDvL9QZ5jKxKpuxKrZeh0duayGkDMg6dfNvvO1lC+i8z6SH6hut2rDwRvmGZcveeWhSnDMi7VJFT/WgVe7CRJbx/OxSNvOvlVViTDI+47c0BeZkXR+QC/wRBsZNRLc4+X07FNpha/XtYSEwaCUO6wIyiN6T55JOkFmNqp9fBEHWG04LfY1N2/IYM= ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:54:26.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T22:54:26.894 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:54:28.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:28.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:28.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:29.342 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T22:54:29.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T22:54:29.695 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:54:29.917 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:29 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3597927331' entity='client.admin' 2024-03-11T22:54:29.917 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:29.917 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:32.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:32 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:32.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:32 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:32.899 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:32 smithi107 ceph-mon[25849]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:32.900 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:32 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:32.900 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:32 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:32.900 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:32 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:32.900 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:32 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:54:33.250 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi131 2024-03-11T22:54:33.250 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:54:33.250 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:54:33.267 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:54:33.268 DEBUG:teuthology.orchestra.run.smithi131:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:33.324 INFO:tasks.cephadm:Adding host smithi131 to orchestrator... 2024-03-11T22:54:33.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch host add smithi131 2024-03-11T22:54:34.076 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:34.583 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:36.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:36 smithi107 ceph-mon[25849]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi131", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:37.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:37 smithi107 ceph-mon[25849]: Deploying cephadm binary to smithi131 2024-03-11T22:54:37.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:37 smithi107 ceph-mon[25849]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:40.109 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi131' with addr '172.21.15.131' 2024-03-11T22:54:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:40 smithi107 ceph-mon[25849]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:40 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:40.425 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:40 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:40.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:54:40.952 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:54:41.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:41 smithi107 ceph-mon[25849]: Added host smithi131 2024-03-11T22:54:41.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:41 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:41.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:41 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:41.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:41 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:41 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:54:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:41 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:41.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:41 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:54:42.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:42 smithi107 ceph-mon[25849]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:54:42.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:42 smithi107 ceph-mon[25849]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:42.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:42 smithi107 ceph-mon[25849]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:54:42.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:42 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:42.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:42 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:42.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:42 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]: dispatch 2024-03-11T22:54:42.363 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:42 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]': finished 2024-03-11T22:54:42.794 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:42.794 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}] 2024-03-11T22:54:43.398 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:43 smithi107 ceph-mon[25849]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:43.398 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:43 smithi107 ceph-mon[25849]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:54:43.398 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:43 smithi107 ceph-mon[25849]: Deploying daemon agent.smithi131 on smithi131 2024-03-11T22:54:43.398 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:43.398 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:43.398 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:43.423 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi195 2024-03-11T22:54:43.423 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:54:43.423 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.conf 2024-03-11T22:54:43.441 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:54:43.442 DEBUG:teuthology.orchestra.run.smithi195:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:43.498 INFO:tasks.cephadm:Adding host smithi195 to orchestrator... 2024-03-11T22:54:43.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch host add smithi195 2024-03-11T22:54:43.730 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:44.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:44.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:54:44.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:44.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:44.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:44.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:44 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:46.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:46 smithi107 ceph-mon[25849]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:47.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:47 smithi107 ceph-mon[25849]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:54:47.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:47 smithi107 ceph-mon[25849]: Deploying cephadm binary to smithi195 2024-03-11T22:54:47.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:47 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:47.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:47 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:47.567 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:47 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:48.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:48 smithi107 ceph-mon[25849]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:48.351 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:48 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:48.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:48 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:48.352 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:48 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:49.880 INFO:teuthology.orchestra.run.smithi107.stdout:Added host 'smithi195' with addr '172.21.15.195' 2024-03-11T22:54:50.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:50 smithi107 ceph-mon[25849]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:50.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:50 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:50.570 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:50 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:50.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T22:54:51.162 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:54:51.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:51 smithi107 ceph-mon[25849]: Added host smithi195 2024-03-11T22:54:51.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:51.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:51.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:51.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:54:51.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:51.299 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:54:52.313 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:52 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]': finished 2024-03-11T22:54:53.082 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:54:53.082 INFO:teuthology.orchestra.run.smithi107.stdout:[{"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.131", "hostname": "smithi131", "labels": [], "status": ""}, {"addr": "172.21.15.195", "hostname": "smithi195", "labels": [], "status": ""}] 2024-03-11T22:54:53.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:53 smithi107 ceph-mon[25849]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:54:53.764 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T22:54:53.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T22:54:54.003 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:54:54.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: Deploying daemon agent.smithi195 on smithi195 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:54.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:55 smithi107 ceph-mon[25849]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:56.366 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2024-03-11T22:54:56.691 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:56 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/2623566915' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:54:56.980 INFO:tasks.cephadm:Adding mon.a on smithi107 2024-03-11T22:54:56.980 INFO:tasks.cephadm:Adding mon.b on smithi131 2024-03-11T22:54:56.980 INFO:tasks.cephadm:Adding mon.c on smithi195 2024-03-11T22:54:56.981 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c' 2024-03-11T22:54:57.217 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:54:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:57 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/2623566915' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:54:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:57 smithi107 ceph-mon[25849]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:54:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:57 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:57 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:57.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:57 smithi107 ceph-mon[25849]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:54:57.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:57 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:54:58.907 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mon update... 2024-03-11T22:54:59.465 DEBUG:teuthology.orchestra.run.smithi131:mon.b> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T22:54:59.468 DEBUG:teuthology.orchestra.run.smithi195:mon.c> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T22:54:59.470 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:54:59.470 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:00.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:00.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c;count:3 2024-03-11T22:55:00.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:00.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:00.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:00.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:00.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:00.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:00.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:00.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: Deploying daemon mon.c on smithi195 2024-03-11T22:55:00.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:54:59 smithi107 ceph-mon[25849]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:00.515 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:02.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:02 smithi107 ceph-mon[25849]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.379 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:55:03.379 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":1,"fsid":"eb9eeed0-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:53:18.742993Z","created":"2024-03-11T22:53:18.742993Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T22:55:03.381 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 1 2024-03-11T22:55:03.468 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:55:03.468 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:55:03.468 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** DB Stats ** 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** Compaction Stats [default] ** 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:55:03.469 INFO:journalctl@ceph.mon.c.smithi195.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** Compaction Stats [default] ** 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:55:03.470 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: Block cache BinnedLRUCache@0x55f1bd9a5090#6 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-03-11T22:55:03.471 INFO:journalctl@ceph.mon.c.smithi195.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:55:03.472 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:55:03.472 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: using public_addr v2:172.21.15.195:0/0 -> [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2024-03-11T22:55:03.472 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: starting mon.c rank -1 at public addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] at bind addrs [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(???) e0 preinit fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout: e1 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout: legacy client fscid: -1 2024-03-11T22:55:03.816 INFO:journalctl@ceph.mon.c.smithi195.stdout: 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout: No filesystems configured 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:03.817 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Added host smithi131 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]: dispatch 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]': finished 2024-03-11T22:55:03.818 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Deploying daemon agent.smithi131 on smithi131 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.819 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Deploying cephadm binary to smithi195 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.820 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.821 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.821 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.821 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:03.821 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Added host smithi195 2024-03-11T22:55:03.821 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.821 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.821 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]': finished 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:03.822 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Deploying daemon agent.smithi195 on smithi195 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.823 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/2623566915' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/2623566915' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c;count:3 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:03.824 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: Deploying daemon mon.c on smithi195 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 ceph-mon[27640]: mon.c@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 bash[27590]: ac7e9e7255a6b5bd46b058e30b57d04ccd3fedc23ec9a7882c6028bbeccd5c01 2024-03-11T22:55:03.825 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:03 smithi195 systemd[1]: Started Ceph mon.c for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:55:05.059 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:55:05.059 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:05.303 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:55:08.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:55:08.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: mon.a calling monitor election 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: mon.c calling monitor election 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: monmap e2: 2 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: fsmap 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:08.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: mgrmap e13: y(active, since 51s) 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: overall HEALTH_OK 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:08.704 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:08 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:09.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:55:09.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: mon.a calling monitor election 2024-03-11T22:55:09.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:09.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:09.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:09.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: mon.c calling monitor election 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: monmap e2: 2 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: fsmap 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: mgrmap e13: y(active, since 51s) 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: overall HEALTH_OK 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:09.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:09.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:08 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:09.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:09 smithi107 ceph-mon[25849]: Deploying daemon mon.b on smithi131 2024-03-11T22:55:09.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:09 smithi107 ceph-mon[25849]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:09.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:09 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:10.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:09 smithi195 ceph-mon[27640]: Deploying daemon mon.b on smithi131 2024-03-11T22:55:10.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:09 smithi195 ceph-mon[27640]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:10.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:09 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:10.080 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:55:10.080 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":2,"fsid":"eb9eeed0-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:55:03.490389Z","created":"2024-03-11T22:53:18.742993Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:55:10.082 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 2 2024-03-11T22:55:10.676 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:10 smithi195 ceph-mon[27640]: from='client.? 172.21.15.195:0/1524885236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:10 smithi107 ceph-mon[25849]: from='client.? 172.21.15.195:0/1524885236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:11.702 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T22:55:11.703 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T22:55:11.950 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:55:11.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:11 smithi107 ceph-mon[25849]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:12.073 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:11 smithi195 ceph-mon[27640]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:12.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 podman[27505]: 2024-03-11 22:55:12.365446448 +0000 UTC m=+0.017836308 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 podman[27505]: 2024-03-11 22:55:12.623577272 +0000 UTC m=+0.275967142 container init 6b411d0d6e6d2e89d9e1aed47ce930d775ecb45d4c5bc786df073cc816b13be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-94a5397) 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 podman[27505]: 2024-03-11 22:55:12.629372352 +0000 UTC m=+0.281762211 container start 6b411d0d6e6d2e89d9e1aed47ce930d775ecb45d4c5bc786df073cc816b13be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: ceph version 19.0.0-1607-g94a53970 (94a539707733325322964c6a187c6a1b41be1daf) squid (dev), process ceph-mon, pid 7 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pidfile_write: ignore empty --pid-file 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: load: jerasure load: lrc 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: RocksDB version: 7.9.2 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Git sha 0 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Compile date 2024-03-08 02:48:09 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: DB SUMMARY 2024-03-11T22:55:13.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: DB Session ID: 9ER8YIL2XX9OW9GOL5HD 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: CURRENT file: CURRENT 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.error_if_exists: 0 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.create_if_missing: 0 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.paranoid_checks: 1 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T22:55:13.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.env: 0x560358069ae0 2024-03-11T22:55:13.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T22:55:13.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.info_log: 0x5603593b38c0 2024-03-11T22:55:13.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T22:55:13.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.statistics: (nil) 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.use_fsync: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_log_file_size: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.allow_fallocate: 1 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.use_direct_reads: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.db_log_dir: 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.wal_dir: 2024-03-11T22:55:13.065 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.write_buffer_manager: 0x560359387400 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T22:55:13.066 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.unordered_write: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.row_cache: None 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.wal_filter: None 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.two_write_queues: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T22:55:13.067 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.wal_compression: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.atomic_flush: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.log_readahead_size: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_background_jobs: 2 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_background_compactions: -1 2024-03-11T22:55:13.068 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_subcompactions: 1 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_open_files: -1 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_background_flushes: -1 2024-03-11T22:55:13.069 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Compression algorithms supported: 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kZSTD supported: 0 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kXpressCompression supported: 0 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kZlibCompression supported: 1 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kSnappyCompression supported: 1 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kLZ4Compression supported: 1 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: kBZip2Compression supported: 0 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T22:55:13.070 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.merge_operator: 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_filter: None 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5603593b39c0) 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: cache_index_and_filter_blocks: 1 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: pin_top_level_index_and_filter: 1 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: index_type: 0 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: data_block_index_type: 0 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: index_shortening: 1 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T22:55:13.071 INFO:journalctl@ceph.mon.b.smithi131.stdout: checksum: 4 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: no_block_cache: 0 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_cache: 0x5603593b7090 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_cache_name: BinnedLRUCache 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_cache_options: 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: capacity : 536870912 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: num_shard_bits : 4 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: strict_capacity_limit : 0 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: high_pri_pool_ratio: 0.000 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_cache_compressed: (nil) 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: persistent_cache: (nil) 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_size: 4096 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_size_deviation: 10 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_restart_interval: 16 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: index_block_restart_interval: 1 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: metadata_block_size: 4096 2024-03-11T22:55:13.072 INFO:journalctl@ceph.mon.b.smithi131.stdout: partition_filters: 0 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: use_delta_encoding: 1 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: filter_policy: bloomfilter 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: whole_key_filtering: 1 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: verify_compression: 0 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: read_amp_bytes_per_bit: 0 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: format_version: 5 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: enable_index_compression: 1 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: block_align: 0 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: max_auto_readahead_size: 262144 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: prepopulate_block_cache: 0 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: initial_auto_readahead_size: 8192 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression: NoCompression 2024-03-11T22:55:13.073 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.num_levels: 7 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T22:55:13.074 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T22:55:13.075 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T22:55:13.076 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.table_properties_collectors: 2024-03-11T22:55:13.077 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.inplace_update_support: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.bloom_locality: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.max_successive_merges: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.ttl: 2592000 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T22:55:13.078 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.enable_blob_files: false 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.min_blob_size: 0 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8f4ee4dd-3a5d-4b7d-b4a8-b3ac9dc9bd7e 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197712658616, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T22:55:13.079 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197712659042, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710197712, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8f4ee4dd-3a5d-4b7d-b4a8-b3ac9dc9bd7e", "db_session_id": "9ER8YIL2XX9OW9GOL5HD", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710197712659122, "job": 1, "event": "recovery_finished"} 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5603594a8000 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: DB pointer 0x560359492000 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout: ** DB Stats ** 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:55:13.080 INFO:journalctl@ceph.mon.b.smithi131.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: ** Compaction Stats [default] ** 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: ** Compaction Stats [default] ** 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: 2024-03-11T22:55:13.081 INFO:journalctl@ceph.mon.b.smithi131.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: Block cache BinnedLRUCache@0x5603593b7090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T22:55:13.082 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: using public_addr v2:172.21.15.131:0/0 -> [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: starting mon.b rank -1 at public addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] at bind addrs [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(???) e0 preinit fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).mds e1 new map 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout: e1 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout: legacy client fscid: -1 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout: 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout: No filesystems configured 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T22:55:13.083 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Added host smithi131 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:13.084 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]: dispatch 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]': finished 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:13.085 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:13.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Deploying daemon agent.smithi131 on smithi131 2024-03-11T22:55:13.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:55:13.086 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.087 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi195", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Deploying cephadm binary to smithi195 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.088 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Added host smithi195 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:55:13.089 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]': finished 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Deploying daemon agent.smithi195 on smithi195 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:55:13.090 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.091 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.091 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.091 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.091 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:13.092 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.092 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:13.092 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.092 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:13.092 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.092 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/2623566915' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/2623566915' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.093 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Saving service mon spec with placement smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c;count:3 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Deploying daemon mon.c on smithi195 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.a calling monitor election 2024-03-11T22:55:13.094 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.c calling monitor election 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: monmap e2: 2 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: fsmap 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mgrmap e13: y(active, since 51s) 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: overall HEALTH_OK 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.095 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: Deploying daemon mon.b on smithi131 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: from='client.? 172.21.15.195:0/1524885236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:13.096 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:13.097 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 ceph-mon[27544]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-11T22:55:13.097 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 bash[27505]: 6b411d0d6e6d2e89d9e1aed47ce930d775ecb45d4c5bc786df073cc816b13be0 2024-03-11T22:55:13.097 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:12 smithi131 systemd[1]: Started Ceph mon.b for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: mon.a calling monitor election 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: mon.c calling monitor election 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: fsmap 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: mgrmap e13: y(active, since 60s) 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: mon.b (rank 2) addr [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] is down (out of quorum) 2024-03-11T22:55:18.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.081 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.146 INFO:teuthology.orchestra.run.smithi195.stdout: 2024-03-11T22:55:18.146 INFO:teuthology.orchestra.run.smithi195.stdout:{"epoch":3,"fsid":"eb9eeed0-dff9-11ee-95c8-87774f69a715","modified":"2024-03-11T22:55:12.682163Z","created":"2024-03-11T22:53:18.742993Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:3300","nonce":0},{"type":"v1","addr":"172.21.15.195:6789","nonce":0}]},"addr":"172.21.15.195:6789/0","public_addr":"172.21.15.195:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:3300","nonce":0},{"type":"v1","addr":"172.21.15.131:6789","nonce":0}]},"addr":"172.21.15.131:6789/0","public_addr":"172.21.15.131:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T22:55:18.147 INFO:teuthology.orchestra.run.smithi195.stderr:dumped monmap epoch 3 2024-03-11T22:55:18.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:55:18.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: mon.a calling monitor election 2024-03-11T22:55:18.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: mon.c calling monitor election 2024-03-11T22:55:18.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:18.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:18.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: fsmap 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: mgrmap e13: y(active, since 60s) 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:55:18.203 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: mon.b (rank 2) addr [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] is down (out of quorum) 2024-03-11T22:55:18.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:18.204 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='client.? 172.21.15.195:0/1226221267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:19.042 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:19.043 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:18 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:19.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:19.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:19.175 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='client.? 172.21.15.195:0/1226221267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.176 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:19.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:19.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:19.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:19.177 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:18 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: mon.a calling monitor election 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: mon.c calling monitor election 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:20.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: fsmap 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: mgrmap e13: y(active, since 60s) 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: mon.b (rank 2) addr [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] is down (out of quorum) 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='client.? 172.21.15.195:0/1226221267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:55:20.063 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:20.064 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:20.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: mon.b calling monitor election 2024-03-11T22:55:20.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: mon.b calling monitor election 2024-03-11T22:55:20.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: mon.a calling monitor election 2024-03-11T22:55:20.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: mon.c calling monitor election 2024-03-11T22:55:20.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:55:20.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:20.225 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: fsmap 2024-03-11T22:55:20.225 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:20.225 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: mgrmap e13: y(active, since 62s) 2024-03-11T22:55:20.225 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:55:20.225 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: Cluster is now healthy 2024-03-11T22:55:20.225 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:20 smithi195 ceph-mon[27640]: overall HEALTH_OK 2024-03-11T22:55:20.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: mon.b calling monitor election 2024-03-11T22:55:20.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: mon.b calling monitor election 2024-03-11T22:55:20.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: mon.a calling monitor election 2024-03-11T22:55:20.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: mon.c calling monitor election 2024-03-11T22:55:20.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:55:20.231 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:20.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: fsmap 2024-03-11T22:55:20.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:20.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: mgrmap e13: y(active, since 62s) 2024-03-11T22:55:20.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:55:20.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: Cluster is now healthy 2024-03-11T22:55:20.232 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:20 smithi107 ceph-mon[25849]: overall HEALTH_OK 2024-03-11T22:55:20.291 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T22:55:20.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: mon.b calling monitor election 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: mon.b calling monitor election 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: mon.a calling monitor election 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: mon.c calling monitor election 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: monmap e3: 3 mons at {a=[v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0],b=[v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0],c=[v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: fsmap 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T22:55:20.366 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: mgrmap e13: y(active, since 62s) 2024-03-11T22:55:20.367 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-11T22:55:20.367 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: Cluster is now healthy 2024-03-11T22:55:20.367 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:20 smithi131 ceph-mon[27544]: overall HEALTH_OK 2024-03-11T22:55:20.528 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:55:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:55:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: Reconfiguring daemon mon.b on smithi131 2024-03-11T22:55:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: Reconfiguring daemon mon.b on smithi131 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.670 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: Reconfiguring mon.b (monmap changed)... 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: Reconfiguring daemon mon.b on smithi131 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:21.671 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.402 INFO:teuthology.orchestra.run.smithi107.stdout:# minimal ceph.conf for eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:22.402 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-03-11T22:55:22.403 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:55:22.403 INFO:teuthology.orchestra.run.smithi107.stdout: mon_host = [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.131:3300/0,v1:172.21.15.131:6789/0] [v2:172.21.15.195:3300/0,v1:172.21.15.195:6789/0] 2024-03-11T22:55:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:22 smithi131 ceph-mon[27544]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:22 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:22 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:22 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:22 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:22 smithi195 ceph-mon[27640]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:22 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:22 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:22 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:22 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:22 smithi107 ceph-mon[25849]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:22.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:22 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:22 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:22 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:22.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:22 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:23.019 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T22:55:23.019 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:55:23.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:55:23.048 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:55:23.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:23.115 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:55:23.115 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:55:23.151 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:55:23.151 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:23.221 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:55:23.221 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T22:55:23.251 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:55:23.251 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:55:23.318 INFO:tasks.cephadm:Adding mgr.y on smithi107 2024-03-11T22:55:23.318 INFO:tasks.cephadm:Adding mgr.x on smithi131 2024-03-11T22:55:23.319 INFO:tasks.cephadm:Adding mgr.z on smithi195 2024-03-11T22:55:23.319 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch apply mgr '3;smithi107=y;smithi131=x;smithi195=z' 2024-03-11T22:55:23.380 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:23 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/263294444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:23.380 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:23 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:23.380 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:23 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:23.381 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:23 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:23.381 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:23 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:23.381 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:23 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:23.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:23 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/263294444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:23.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:23 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:23.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:23 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:23.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:23 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:23.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:23 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:23.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:23 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:23.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:23 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/263294444' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:23.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:23 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:23.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:23 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:23.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:23 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T22:55:23.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:23 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T22:55:23.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:23 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:24.203 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: Reconfiguring daemon mon.c on smithi195 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:24.425 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:24.426 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:55:24.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: Reconfiguring daemon mon.c on smithi195 2024-03-11T22:55:24.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:24.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:24.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:24.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:24.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: Reconfiguring mon.c (monmap changed)... 2024-03-11T22:55:24.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: Reconfiguring daemon mon.c on smithi195 2024-03-11T22:55:24.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:24.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:24.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:24.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:24.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:24.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:25.764 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled mgr update... 2024-03-11T22:55:26.290 DEBUG:teuthology.orchestra.run.smithi131:mgr.x> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T22:55:26.293 DEBUG:teuthology.orchestra.run.smithi195:mgr.z> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.z.service 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:26.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:26.331 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T22:55:26.331 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:55:26.332 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:55:26.350 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:55:26.351 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/nvme1n1 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/nvme1n1 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 764 Links: 1 Device type: 103,2 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:55:08.985628471 +0000 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:34.763298280 +0000 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:34.763298280 +0000 2024-03-11T22:55:26.408 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:34.755298416 +0000 2024-03-11T22:55:26.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:55:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:55:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:55:26.479 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000263096 s, 1.9 MB/s 2024-03-11T22:55:26.480 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:55:26.537 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/nvme2n1 2024-03-11T22:55:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:26.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:26.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:26.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/nvme2n1 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 774 Links: 1 Device type: 103,4 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:55:08.996628280 +0000 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:34.917295651 +0000 2024-03-11T22:55:26.594 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:34.917295651 +0000 2024-03-11T22:55:26.595 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:34.905295856 +0000 2024-03-11T22:55:26.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:55:26.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:26.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:26.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:26.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:26.659 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:26.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:26.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:26.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:26.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:26.660 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:26.661 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:55:26.662 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:55:26.662 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000254985 s, 2.0 MB/s 2024-03-11T22:55:26.663 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:55:26.720 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/nvme3n1 2024-03-11T22:55:26.775 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/nvme3n1 2024-03-11T22:55:26.775 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:26.776 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T22:55:26.776 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:26.776 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:26.776 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:55:09.007628089 +0000 2024-03-11T22:55:26.776 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:35.063293159 +0000 2024-03-11T22:55:26.776 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:35.063293159 +0000 2024-03-11T22:55:26.776 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:35.052293346 +0000 2024-03-11T22:55:26.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:55:26.842 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:55:26.842 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:55:26.842 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000230215 s, 2.2 MB/s 2024-03-11T22:55:26.843 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:55:26.900 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/nvme4n1 2024-03-11T22:55:26.955 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/nvme4n1 2024-03-11T22:55:26.955 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:26.955 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:55:26.955 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:26.955 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:26.956 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-03-11 22:55:09.017627916 +0000 2024-03-11T22:55:26.956 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-03-11 22:51:35.206290718 +0000 2024-03-11T22:55:26.956 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-03-11 22:51:35.206290718 +0000 2024-03-11T22:55:26.956 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-03-11 22:51:35.200290820 +0000 2024-03-11T22:55:26.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:55:27.020 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-03-11T22:55:27.021 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-03-11T22:55:27.021 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000192295 s, 2.7 MB/s 2024-03-11T22:55:27.022 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:55:27.078 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:55:27.078 DEBUG:teuthology.orchestra.run.smithi131:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:55:27.097 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:55:27.097 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme1n1 2024-03-11T22:55:27.155 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme1n1 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:55:20.200344594 +0000 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:36.422285422 +0000 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:36.422285422 +0000 2024-03-11T22:55:27.156 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:36.409285709 +0000 2024-03-11T22:55:27.157 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:55:27.224 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:55:27.225 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:55:27.225 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000344602 s, 1.5 MB/s 2024-03-11T22:55:27.226 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:55:27.284 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme2n1 2024-03-11T22:55:27.340 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme2n1 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:55:20.211344355 +0000 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:36.579281963 +0000 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:36.579281963 +0000 2024-03-11T22:55:27.341 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:36.562282338 +0000 2024-03-11T22:55:27.341 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:55:27.407 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:55:27.408 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:55:27.408 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000220694 s, 2.3 MB/s 2024-03-11T22:55:27.409 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:55:27.466 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme3n1 2024-03-11T22:55:27.521 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme3n1 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:55:20.222344115 +0000 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:36.680279738 +0000 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:36.680279738 +0000 2024-03-11T22:55:27.522 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:36.668280003 +0000 2024-03-11T22:55:27.522 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:55:27.539 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:27 smithi195 ceph-mon[27640]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=y;smithi131=x;smithi195=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:27.539 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:27 smithi195 ceph-mon[27640]: Saving service mgr spec with placement smithi107=y;smithi131=x;smithi195=z;count:3 2024-03-11T22:55:27.539 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:27 smithi195 ceph-mon[27640]: Deploying daemon mgr.z on smithi195 2024-03-11T22:55:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:27 smithi131 ceph-mon[27544]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=y;smithi131=x;smithi195=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:27 smithi131 ceph-mon[27544]: Saving service mgr spec with placement smithi107=y;smithi131=x;smithi195=z;count:3 2024-03-11T22:55:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:27 smithi131 ceph-mon[27544]: Deploying daemon mgr.z on smithi195 2024-03-11T22:55:27.586 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:55:27.586 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:55:27.586 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000201906 s, 2.5 MB/s 2024-03-11T22:55:27.587 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:55:27.643 DEBUG:teuthology.orchestra.run.smithi131:> stat /dev/nvme4n1 2024-03-11T22:55:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: File: /dev/nvme4n1 2024-03-11T22:55:27.698 INFO:teuthology.orchestra.run.smithi131.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:27.698 INFO:teuthology.orchestra.run.smithi131.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:55:27.698 INFO:teuthology.orchestra.run.smithi131.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:27.698 INFO:teuthology.orchestra.run.smithi131.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:27.698 INFO:teuthology.orchestra.run.smithi131.stdout:Access: 2024-03-11 22:55:20.233343875 +0000 2024-03-11T22:55:27.698 INFO:teuthology.orchestra.run.smithi131.stdout:Modify: 2024-03-11 22:51:36.837276279 +0000 2024-03-11T22:55:27.699 INFO:teuthology.orchestra.run.smithi131.stdout:Change: 2024-03-11 22:51:36.837276279 +0000 2024-03-11T22:55:27.699 INFO:teuthology.orchestra.run.smithi131.stdout: Birth: 2024-03-11 22:51:36.829276456 +0000 2024-03-11T22:55:27.699 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:55:27.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:27 smithi107 ceph-mon[25849]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi107=y;smithi131=x;smithi195=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:27.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:27 smithi107 ceph-mon[25849]: Saving service mgr spec with placement smithi107=y;smithi131=x;smithi195=z;count:3 2024-03-11T22:55:27.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:27 smithi107 ceph-mon[25849]: Deploying daemon mgr.z on smithi195 2024-03-11T22:55:27.764 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records in 2024-03-11T22:55:27.764 INFO:teuthology.orchestra.run.smithi131.stderr:1+0 records out 2024-03-11T22:55:27.764 INFO:teuthology.orchestra.run.smithi131.stderr:512 bytes copied, 0.000203946 s, 2.5 MB/s 2024-03-11T22:55:27.765 DEBUG:teuthology.orchestra.run.smithi131:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:55:27.820 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:55:27.821 DEBUG:teuthology.orchestra.run.smithi195:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T22:55:27.838 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-03-11T22:55:27.838 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/nvme1n1 2024-03-11T22:55:27.895 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/nvme1n1 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:55:20.169095124 +0000 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:38.070147208 +0000 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:38.070147208 +0000 2024-03-11T22:55:27.896 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:38.060147390 +0000 2024-03-11T22:55:27.896 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-03-11T22:55:27.963 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:55:27.964 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:55:27.964 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000131649 s, 3.9 MB/s 2024-03-11T22:55:27.965 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-03-11T22:55:28.021 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/nvme2n1 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/nvme2n1 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:55:20.177094976 +0000 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:38.227144348 +0000 2024-03-11T22:55:28.075 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:38.227144348 +0000 2024-03-11T22:55:28.076 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:38.217144530 +0000 2024-03-11T22:55:28.076 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-03-11T22:55:28.142 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:55:28.143 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:55:28.143 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000110122 s, 4.6 MB/s 2024-03-11T22:55:28.144 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-03-11T22:55:28.198 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/nvme3n1 2024-03-11T22:55:28.254 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/nvme3n1 2024-03-11T22:55:28.254 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:28.254 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 788 Links: 1 Device type: 103,6 2024-03-11T22:55:28.254 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:28.255 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:28.255 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:55:20.186094810 +0000 2024-03-11T22:55:28.255 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:38.351142089 +0000 2024-03-11T22:55:28.255 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:38.351142089 +0000 2024-03-11T22:55:28.255 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:38.343142235 +0000 2024-03-11T22:55:28.255 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-03-11T22:55:28.321 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:55:28.322 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:55:28.322 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000121218 s, 4.2 MB/s 2024-03-11T22:55:28.323 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-03-11T22:55:28.379 DEBUG:teuthology.orchestra.run.smithi195:> stat /dev/nvme4n1 2024-03-11T22:55:28.434 INFO:teuthology.orchestra.run.smithi195.stdout: File: /dev/nvme4n1 2024-03-11T22:55:28.434 INFO:teuthology.orchestra.run.smithi195.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-03-11T22:55:28.434 INFO:teuthology.orchestra.run.smithi195.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-03-11T22:55:28.434 INFO:teuthology.orchestra.run.smithi195.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-03-11T22:55:28.435 INFO:teuthology.orchestra.run.smithi195.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-03-11T22:55:28.435 INFO:teuthology.orchestra.run.smithi195.stdout:Access: 2024-03-11 22:55:20.194094661 +0000 2024-03-11T22:55:28.435 INFO:teuthology.orchestra.run.smithi195.stdout:Modify: 2024-03-11 22:51:38.508139228 +0000 2024-03-11T22:55:28.435 INFO:teuthology.orchestra.run.smithi195.stdout:Change: 2024-03-11 22:51:38.508139228 +0000 2024-03-11T22:55:28.435 INFO:teuthology.orchestra.run.smithi195.stdout: Birth: 2024-03-11 22:51:38.498139411 +0000 2024-03-11T22:55:28.435 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-03-11T22:55:28.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:28 smithi107 ceph-mon[25849]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:28.497 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:28 smithi195 ceph-mon[27640]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:28.497 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:28 smithi195 systemd[1]: Started Ceph mgr.z for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:55:28.500 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records in 2024-03-11T22:55:28.500 INFO:teuthology.orchestra.run.smithi195.stderr:1+0 records out 2024-03-11T22:55:28.500 INFO:teuthology.orchestra.run.smithi195.stderr:512 bytes copied, 0.000107853 s, 4.7 MB/s 2024-03-11T22:55:28.501 DEBUG:teuthology.orchestra.run.smithi195:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-03-11T22:55:28.556 INFO:tasks.cephadm:Deploying osd.0 on smithi107 with /dev/nvme4n1... 2024-03-11T22:55:28.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:55:28.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:28 smithi131 ceph-mon[27544]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:28.810 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:55:29.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:28 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:28.791+0000 7f1f5a842200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:55:29.079 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:28 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:28.884+0000 7f1f5a842200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: Deploying daemon mgr.x on smithi131 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:29 smithi107 ceph-mon[25849]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:29.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: Deploying daemon mgr.x on smithi131 2024-03-11T22:55:29.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:29 smithi131 ceph-mon[27544]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:29.525+0000 7f1f5a842200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:29.616+0000 7f1f5a842200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:29.758+0000 7f1f5a842200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:29.764 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:29.765 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: Deploying daemon mgr.x on smithi131 2024-03-11T22:55:29.765 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.765 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.765 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:29.765 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-mon[27640]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:30.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:29.854+0000 7f1f5a842200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:55:30.079 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:29.935+0000 7f1f5a842200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:55:30.491 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:30 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:30.198+0000 7f1f5a842200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:55:30.491 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:30 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:30.307+0000 7f1f5a842200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:55:30.491 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:30 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:30.392+0000 7f1f5a842200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:55:30.491 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:30 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:30.486+0000 7f1f5a842200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:55:30.644 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 systemd[1]: Starting Ceph mgr.x for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:55:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:55:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:55:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:55:30.689 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-03-11T22:55:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-03-11T22:55:30.689 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130666 s, 802 MB/s 2024-03-11T22:55:30.689 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-03-11T22:55:30.828 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:30 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:30.663+0000 7f1f5a842200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:55:30.929 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 podman[29226]: 2024-03-11 22:55:30.639355432 +0000 UTC m=+0.106289530 container create 2df2ccf01faa260d3c21645299b09774a5b2ba5927ee18becc1ded62a4246d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397) 2024-03-11T22:55:30.929 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 podman[29226]: 2024-03-11 22:55:30.550445753 +0000 UTC m=+0.017379783 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:55:30.929 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 podman[29226]: 2024-03-11 22:55:30.816770638 +0000 UTC m=+0.283704661 container init 2df2ccf01faa260d3c21645299b09774a5b2ba5927ee18becc1ded62a4246d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T22:55:30.929 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 podman[29226]: 2024-03-11 22:55:30.821950599 +0000 UTC m=+0.288884618 container start 2df2ccf01faa260d3c21645299b09774a5b2ba5927ee18becc1ded62a4246d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:55:30.929 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 ceph-mgr[29265]: -- 172.21.15.131:0/2436930392 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5594b161a1a0 con 0x5594b15f0c00 2024-03-11T22:55:30.930 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 bash[29226]: 2df2ccf01faa260d3c21645299b09774a5b2ba5927ee18becc1ded62a4246d46 2024-03-11T22:55:31.299 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:30 smithi131 systemd[1]: Started Ceph mgr.x for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:55:31.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi107:/dev/nvme4n1 2024-03-11T22:55:31.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:31.380+0000 7f3ea9ca9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:55:31.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:31.479+0000 7f3ea9ca9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:55:31.567 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:55:31.791 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:31.657+0000 7f1f5a842200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:55:32.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:31.786+0000 7f1f5a842200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:32.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:55:32.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:32.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:32.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:32.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:31 smithi195 ceph-mon[27640]: Reconfiguring daemon mgr.y on smithi107 2024-03-11T22:55:32.107 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.107 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.107 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.107 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.107 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:32.107 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:32.107 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:32.108 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:31 smithi131 ceph-mon[27544]: Reconfiguring daemon mgr.y on smithi107 2024-03-11T22:55:32.387 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:32.177+0000 7f3ea9ca9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:55:32.387 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:32.274+0000 7f3ea9ca9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:55:32.412 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T22:55:32.413 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:32.414 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:31 smithi107 ceph-mon[25849]: Reconfiguring daemon mgr.y on smithi107 2024-03-11T22:55:32.452 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:32.134+0000 7f1f5a842200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:55:32.452 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:32.228+0000 7f1f5a842200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:55:32.452 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:32.446+0000 7f1f5a842200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:55:32.729 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:32.428+0000 7f3ea9ca9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:55:32.729 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:32.525+0000 7f3ea9ca9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:55:32.729 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:32.607+0000 7f3ea9ca9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:55:32.730 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:32.642+0000 7f1f5a842200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:55:33.008 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:32.888+0000 7f3ea9ca9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:55:33.009 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:33 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:33.003+0000 7f3ea9ca9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:55:33.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:32.941+0000 7f1f5a842200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:55:33.291 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:33 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:33.094+0000 7f3ea9ca9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:55:33.291 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:33 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:33.191+0000 7f3ea9ca9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:55:33.483 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:33 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:33.483 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:33 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:33.218+0000 7f1f5a842200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:55:33.483 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:33 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:33.478+0000 7f1f5a842200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:55:33.544 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:33 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:33.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:33 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:33.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:33 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:33.383+0000 7f3ea9ca9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:34.405+0000 7f3ea9ca9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:55:34.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:34.535+0000 7f3ea9ca9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:55:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:34.350+0000 7f1f5a842200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.174 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:34.883+0000 7f3ea9ca9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:55:35.174 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:34 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:34.973+0000 7f3ea9ca9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:55:35.174 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:35.169+0000 7f3ea9ca9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:55:35.234 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:34 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:34.920+0000 7f1f5a842200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:55:35.234 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:35.017+0000 7f1f5a842200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:55:35.234 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:55:35.229+0000 7f1f5a842200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:55:35.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:55:35.391 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.392 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: Standby manager daemon z started 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:55:35.393 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:35 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:35.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:35.354+0000 7f3ea9ca9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:55:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:55:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:55:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: Standby manager daemon z started 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:35.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:55:35.562 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:35.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:35.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: Standby manager daemon z started 2024-03-11T22:55:35.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:55:35.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:35.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:55:35.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:35 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2803766464' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:35.893 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:35.629+0000 7f3ea9ca9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:55:35.893 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:35 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:35.888+0000 7f3ea9ca9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:55:36.310 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:36 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:36.115+0000 7f3ea9ca9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:55:36.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:36 smithi107 ceph-mon[25849]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:36.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:36 smithi107 ceph-mon[25849]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:36.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:36 smithi107 ceph-mon[25849]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-11T22:55:36.539 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:36 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:55:36.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:36 smithi195 ceph-mon[27640]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:36.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:36 smithi195 ceph-mon[27640]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:36.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:36 smithi195 ceph-mon[27640]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-11T22:55:36.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:36 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:55:36.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:36 smithi131 ceph-mon[27544]: from='client.14226 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:36.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:36 smithi131 ceph-mon[27544]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:36.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:36 smithi131 ceph-mon[27544]: mgrmap e14: y(active, since 78s), standbys: z 2024-03-11T22:55:36.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:36 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:55:37.310 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:36 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:36.896+0000 7f3ea9ca9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:55:37.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:37 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3807552047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cec53b73-16f2-486e-803c-c603a1502b96"}]: dispatch 2024-03-11T22:55:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:37 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3807552047' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cec53b73-16f2-486e-803c-c603a1502b96"}]': finished 2024-03-11T22:55:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:37 smithi195 ceph-mon[27640]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:55:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:37 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:37.584 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:37 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3807552047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cec53b73-16f2-486e-803c-c603a1502b96"}]: dispatch 2024-03-11T22:55:37.584 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:37 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3807552047' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cec53b73-16f2-486e-803c-c603a1502b96"}]': finished 2024-03-11T22:55:37.584 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:37 smithi107 ceph-mon[25849]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:55:37.584 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:37 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:37.768 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:37 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3807552047' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cec53b73-16f2-486e-803c-c603a1502b96"}]: dispatch 2024-03-11T22:55:37.769 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:37 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3807552047' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cec53b73-16f2-486e-803c-c603a1502b96"}]': finished 2024-03-11T22:55:37.769 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:37 smithi131 ceph-mon[27544]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T22:55:37.769 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:37 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:37.769 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:37 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:37.459+0000 7f3ea9ca9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:55:37.769 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:37 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:37.554+0000 7f3ea9ca9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:55:37.769 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:55:37 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:55:37.763+0000 7f3ea9ca9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:55:38.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:38 smithi195 ceph-mon[27640]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:38 smithi195 ceph-mon[27640]: Standby manager daemon x started 2024-03-11T22:55:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:38 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:55:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:38 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:38 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:55:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:38 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:38.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:38 smithi107 ceph-mon[25849]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:38.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:38 smithi107 ceph-mon[25849]: Standby manager daemon x started 2024-03-11T22:55:38.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:38 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:55:38.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:38 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:38.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:38 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:55:38.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:38 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:38.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:38 smithi131 ceph-mon[27544]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:38.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:38 smithi131 ceph-mon[27544]: Standby manager daemon x started 2024-03-11T22:55:38.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:38 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:55:38.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:38 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:55:38.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:38 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:55:38.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:38 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/3875202017' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:55:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:39 smithi107 ceph-mon[25849]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T22:55:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:39 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:55:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:39 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1434418072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:55:39.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:39 smithi131 ceph-mon[27544]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T22:55:39.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:39 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:55:39.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:39 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1434418072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:55:39.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:39 smithi195 ceph-mon[27640]: mgrmap e15: y(active, since 81s), standbys: z, x 2024-03-11T22:55:39.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:39 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:55:39.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:39 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1434418072' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:55:40.696 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:40 smithi107 ceph-mon[25849]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:40.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:40 smithi131 ceph-mon[27544]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:40.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:40 smithi195 ceph-mon[27640]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:42.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:42 smithi107 ceph-mon[25849]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:42.813 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:42 smithi131 ceph-mon[27544]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:42.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:42 smithi195 ceph-mon[27640]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:44.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:44 smithi107 ceph-mon[25849]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:44.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:44 smithi131 ceph-mon[27544]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:44.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:44 smithi195 ceph-mon[27640]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:45.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:45 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:55:45.496 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:45 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:45.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:45 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:55:45.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:45 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:45.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:45 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T22:55:45.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:45 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:46.450 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:46 smithi107 ceph-mon[25849]: Deploying daemon osd.0 on smithi107 2024-03-11T22:55:46.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:46 smithi107 ceph-mon[25849]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:46 smithi131 ceph-mon[27544]: Deploying daemon osd.0 on smithi107 2024-03-11T22:55:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:46 smithi131 ceph-mon[27544]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:46.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:46 smithi195 ceph-mon[27640]: Deploying daemon osd.0 on smithi107 2024-03-11T22:55:46.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:46 smithi195 ceph-mon[27640]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:48.575 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:48 smithi107 ceph-mon[25849]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:48.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:48 smithi131 ceph-mon[27544]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:48.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:48 smithi195 ceph-mon[27640]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:50.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:49 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:49 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:49 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.068 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:49 smithi107 ceph-mon[25849]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:50.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:49 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:49 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:49 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:49 smithi131 ceph-mon[27544]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:50.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:49 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:49 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:49 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:50.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:49 smithi195 ceph-mon[27640]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:51.326 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.327 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.327 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:51.327 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:51.327 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:51.327 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:51.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:51.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:51.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:51.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:51.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:51.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:51.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:52.430 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:52 smithi195 ceph-mon[27640]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:52.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:52 smithi107 ceph-mon[25849]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:52.687 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:52 smithi131 ceph-mon[27544]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:54.129 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 0 on host 'smithi107' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:54.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:54.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:54.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:54.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:55:54.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.673 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:54.674 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.700 DEBUG:teuthology.orchestra.run.smithi107:osd.0> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T22:55:54.702 INFO:tasks.cephadm:Deploying osd.1 on smithi131 with /dev/nvme4n1... 2024-03-11T22:55:54.703 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:55:54.729 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.730 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:54.731 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:54.731 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:55:54.731 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:55:55.072 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:55:55.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:55 smithi131 ceph-mon[27544]: Adjusting osd_memory_target on smithi107 to 16064M 2024-03-11T22:55:55.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:55 smithi131 ceph-mon[27544]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:55:55.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:55 smithi131 ceph-mon[27544]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:55:55.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:55 smithi131 ceph-mon[27544]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-11T22:55:55.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:55 smithi107 ceph-mon[25849]: Adjusting osd_memory_target on smithi107 to 16064M 2024-03-11T22:55:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:55 smithi107 ceph-mon[25849]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:55:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:55 smithi107 ceph-mon[25849]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:55:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:55 smithi107 ceph-mon[25849]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-11T22:55:55.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:55.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:55 smithi195 ceph-mon[27640]: Adjusting osd_memory_target on smithi107 to 16064M 2024-03-11T22:55:55.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:55 smithi195 ceph-mon[27640]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T22:55:55.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:55 smithi195 ceph-mon[27640]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T22:55:55.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:55 smithi195 ceph-mon[27640]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]: dispatch 2024-03-11T22:55:55.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:56.202 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:55:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0[38174]: 2024-03-11T22:55:55.817+0000 7fa295de1700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T22:55:56.202 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:55:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0[38174]: 2024-03-11T22:55:55.819+0000 7fa28dbcc700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:55:56.457 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:56 smithi131 ceph-mon[27544]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:56.458 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:56 smithi131 ceph-mon[27544]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-11T22:55:56.458 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:56 smithi131 ceph-mon[27544]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:55:56.458 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:56 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:56.458 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:56 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:56.458 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:56 smithi131 ceph-mon[27544]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' 2024-03-11T22:55:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:56 smithi107 ceph-mon[25849]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:56 smithi107 ceph-mon[25849]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-11T22:55:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:56 smithi107 ceph-mon[25849]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:55:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:56 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:56 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:56.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:56 smithi107 ceph-mon[25849]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' 2024-03-11T22:55:56.729 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:56 smithi195 ceph-mon[27640]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:56.730 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:56 smithi195 ceph-mon[27640]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-03-11T22:55:56.730 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:56 smithi195 ceph-mon[27640]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T22:55:56.730 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:56 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:56.730 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:56 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:56.730 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:56 smithi195 ceph-mon[27640]: from='osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137]' entity='osd.0' 2024-03-11T22:55:56.778 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:55:56.778 INFO:teuthology.orchestra.run.smithi131.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:55:56.779 INFO:teuthology.orchestra.run.smithi131.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:55:56.779 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10+0 records in 2024-03-11T22:55:56.779 INFO:teuthology.orchestra.run.smithi131.stderr:10+0 records out 2024-03-11T22:55:56.779 INFO:teuthology.orchestra.run.smithi131.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126858 s, 827 MB/s 2024-03-11T22:55:56.779 INFO:teuthology.orchestra.run.smithi131.stderr:--> Zapping successful for: 2024-03-11T22:55:57.321 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi131:/dev/nvme4n1 2024-03-11T22:55:57.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:57 smithi195 ceph-mon[27640]: purged_snaps scrub starts 2024-03-11T22:55:57.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:57 smithi195 ceph-mon[27640]: purged_snaps scrub ok 2024-03-11T22:55:57.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:57 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:57.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:57 smithi195 ceph-mon[27640]: osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137] boot 2024-03-11T22:55:57.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:57 smithi195 ceph-mon[27640]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:55:57.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:57 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:57.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:57 smithi195 ceph-mon[27640]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:57.597 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:57 smithi107 ceph-mon[25849]: purged_snaps scrub starts 2024-03-11T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:57 smithi107 ceph-mon[25849]: purged_snaps scrub ok 2024-03-11T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:57 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:57 smithi107 ceph-mon[25849]: osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137] boot 2024-03-11T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:57 smithi107 ceph-mon[25849]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:57 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:57.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:57 smithi107 ceph-mon[25849]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:57.711 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:57 smithi131 ceph-mon[27544]: purged_snaps scrub starts 2024-03-11T22:55:57.711 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:57 smithi131 ceph-mon[27544]: purged_snaps scrub ok 2024-03-11T22:55:57.711 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:57 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:57.711 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:57 smithi131 ceph-mon[27544]: osd.0 [v2:172.21.15.107:6802/2877550137,v1:172.21.15.107:6803/2877550137] boot 2024-03-11T22:55:57.712 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:57 smithi131 ceph-mon[27544]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T22:55:57.712 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:57 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:55:57.712 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:57 smithi131 ceph-mon[27544]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T22:55:59.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:59 smithi107 ceph-mon[25849]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:55:59.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:59 smithi107 ceph-mon[25849]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:55:59.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:59 smithi107 ceph-mon[25849]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:59.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:55:59.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:55:59.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:55:59 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:59.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:59 smithi131 ceph-mon[27544]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:55:59.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:59 smithi131 ceph-mon[27544]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:55:59.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:59 smithi131 ceph-mon[27544]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:59.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:59 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:55:59.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:59 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:55:59.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:55:59 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:55:59.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:59 smithi195 ceph-mon[27640]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T22:55:59.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:59 smithi195 ceph-mon[27640]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:55:59.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:59 smithi195 ceph-mon[27640]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi131:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:55:59.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:59 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:55:59.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:59 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:55:59.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:55:59 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:02 smithi131 ceph-mon[27544]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:02 smithi131 ceph-mon[27544]: from='client.? 172.21.15.131:0/2746011000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]: dispatch 2024-03-11T22:56:02.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:02 smithi131 ceph-mon[27544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]: dispatch 2024-03-11T22:56:02.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:02 smithi131 ceph-mon[27544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]': finished 2024-03-11T22:56:02.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:02 smithi131 ceph-mon[27544]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:56:02.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:02.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:02 smithi195 ceph-mon[27640]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:02.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:02 smithi195 ceph-mon[27640]: from='client.? 172.21.15.131:0/2746011000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]: dispatch 2024-03-11T22:56:02.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:02 smithi195 ceph-mon[27640]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]: dispatch 2024-03-11T22:56:02.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:02 smithi195 ceph-mon[27640]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]': finished 2024-03-11T22:56:02.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:02 smithi195 ceph-mon[27640]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:56:02.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:02.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:02 smithi107 ceph-mon[25849]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:02.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:02 smithi107 ceph-mon[25849]: from='client.? 172.21.15.131:0/2746011000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]: dispatch 2024-03-11T22:56:02.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:02 smithi107 ceph-mon[25849]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]: dispatch 2024-03-11T22:56:02.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:02 smithi107 ceph-mon[25849]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18"}]': finished 2024-03-11T22:56:02.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:02 smithi107 ceph-mon[25849]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T22:56:02.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:03 smithi131 ceph-mon[27544]: from='client.? 172.21.15.131:0/1885065795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:03.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:03 smithi195 ceph-mon[27640]: from='client.? 172.21.15.131:0/1885065795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:03.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:03 smithi107 ceph-mon[25849]: from='client.? 172.21.15.131:0/1885065795' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:04.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:04 smithi131 ceph-mon[27544]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:04.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:04 smithi195 ceph-mon[27640]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:04.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:04 smithi107 ceph-mon[25849]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:06.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:06 smithi131 ceph-mon[27544]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:06.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:06 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:06 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:06 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:06 smithi195 ceph-mon[27640]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:06.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:06 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:06 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:06 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:06 smithi107 ceph-mon[25849]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:06.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:06 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:06 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:06.636 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:06 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:07.540 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:07 smithi107 ceph-mon[25849]: Detected new or changed devices on smithi107 2024-03-11T22:56:07.552 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:07 smithi131 ceph-mon[27544]: Detected new or changed devices on smithi107 2024-03-11T22:56:07.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:07 smithi195 ceph-mon[27640]: Detected new or changed devices on smithi107 2024-03-11T22:56:08.399 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:08 smithi131 ceph-mon[27544]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:08.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:08 smithi195 ceph-mon[27640]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:08.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:08 smithi107 ceph-mon[25849]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:09.552 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:09 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:09.552 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:09 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:09.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:09 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:09.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:09 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:09.596 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:09 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:09.596 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:09 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:10.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:10 smithi131 ceph-mon[27544]: Deploying daemon osd.1 on smithi131 2024-03-11T22:56:10.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:10 smithi131 ceph-mon[27544]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:10.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:10 smithi195 ceph-mon[27640]: Deploying daemon osd.1 on smithi131 2024-03-11T22:56:10.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:10 smithi195 ceph-mon[27640]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:10.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:10 smithi107 ceph-mon[25849]: Deploying daemon osd.1 on smithi131 2024-03-11T22:56:10.646 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:10 smithi107 ceph-mon[25849]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:12.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:12 smithi131 ceph-mon[27544]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:12.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:12 smithi195 ceph-mon[27640]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:12.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:12 smithi107 ceph-mon[25849]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.330 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:14 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:14.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:14.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:14 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T22:56:14.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:14.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:14.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:14 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:15.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:15 smithi195 ceph-mon[27640]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T22:56:15.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:15 smithi195 ceph-mon[27640]: Reconfiguring daemon osd.1 on smithi131 2024-03-11T22:56:15.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:15 smithi195 ceph-mon[27640]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:15.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:15 smithi195 ceph-mon[27640]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:15.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:15 smithi195 ceph-mon[27640]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T22:56:15.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:15 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:15.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:15 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:15.334 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:15 smithi131 ceph-mon[27544]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T22:56:15.334 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:15 smithi131 ceph-mon[27544]: Reconfiguring daemon osd.1 on smithi131 2024-03-11T22:56:15.334 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:15 smithi131 ceph-mon[27544]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:15.334 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:15 smithi131 ceph-mon[27544]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:15.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:15 smithi131 ceph-mon[27544]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T22:56:15.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:15 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:15.335 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:15 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:15 smithi107 ceph-mon[25849]: Reconfiguring osd.1 (unknown last config time)... 2024-03-11T22:56:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:15 smithi107 ceph-mon[25849]: Reconfiguring daemon osd.1 on smithi131 2024-03-11T22:56:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:15 smithi107 ceph-mon[25849]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:15 smithi107 ceph-mon[25849]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:15 smithi107 ceph-mon[25849]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-11T22:56:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:15 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:15 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:17.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:17.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:17.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:17.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:17.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T22:56:17.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:17 smithi131 ceph-mon[27544]: Cluster is now healthy 2024-03-11T22:56:17.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:17.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:17.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:17.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:17.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:17.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T22:56:17.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:17 smithi195 ceph-mon[27640]: Cluster is now healthy 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-11T22:56:17.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:17 smithi107 ceph-mon[25849]: Cluster is now healthy 2024-03-11T22:56:18.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:18 smithi131 ceph-mon[27544]: from='osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:18.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:18 smithi131 ceph-mon[27544]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:18.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:18 smithi107 ceph-mon[25849]: from='osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:18.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:18 smithi107 ceph-mon[25849]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:18.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:18 smithi195 ceph-mon[27640]: from='osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:18.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:18 smithi195 ceph-mon[27640]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T22:56:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:19 smithi107 ceph-mon[25849]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:19 smithi107 ceph-mon[25849]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:56:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:19 smithi107 ceph-mon[25849]: from='osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-11T22:56:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:19 smithi107 ceph-mon[25849]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:56:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:19 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:19 smithi107 ceph-mon[25849]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-11T22:56:19.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:19 smithi107 ceph-mon[25849]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-11T22:56:19.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:19 smithi131 ceph-mon[27544]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:19.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:19 smithi131 ceph-mon[27544]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:56:19.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:19 smithi131 ceph-mon[27544]: from='osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-11T22:56:19.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:19 smithi131 ceph-mon[27544]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:56:19.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:19 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:19.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:19 smithi131 ceph-mon[27544]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-11T22:56:19.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:19 smithi131 ceph-mon[27544]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-11T22:56:19.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:19 smithi195 ceph-mon[27640]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:19.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:19 smithi195 ceph-mon[27640]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T22:56:19.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:19 smithi195 ceph-mon[27640]: from='osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-11T22:56:19.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:19 smithi195 ceph-mon[27640]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T22:56:19.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:19 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:19.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:19 smithi195 ceph-mon[27640]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]: dispatch 2024-03-11T22:56:19.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:19 smithi195 ceph-mon[27640]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi131", "root=default"]}]': finished 2024-03-11T22:56:20.250 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:20 smithi131 ceph-mon[27544]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:56:20.251 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:20 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:20.251 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:20 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:20.251 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:20 smithi131 ceph-mon[27544]: from='osd.1 ' entity='osd.1' 2024-03-11T22:56:20.301 INFO:teuthology.orchestra.run.smithi131.stdout:Created osd(s) 1 on host 'smithi131' 2024-03-11T22:56:20.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:20 smithi107 ceph-mon[25849]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:56:20.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:20 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:20.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:20 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:20.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:20 smithi107 ceph-mon[25849]: from='osd.1 ' entity='osd.1' 2024-03-11T22:56:20.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:20 smithi195 ceph-mon[27640]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T22:56:20.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:20 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:20.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:20 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:20.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:20 smithi195 ceph-mon[27640]: from='osd.1 ' entity='osd.1' 2024-03-11T22:56:20.914 DEBUG:teuthology.orchestra.run.smithi131:osd.1> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T22:56:20.916 INFO:tasks.cephadm:Deploying osd.2 on smithi195 with /dev/nvme4n1... 2024-03-11T22:56:20.917 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-03-11T22:56:21.165 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: purged_snaps scrub starts 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: purged_snaps scrub ok 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: Adjusting osd_memory_target on smithi131 to 16064M 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:21.285 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:21.286 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.286 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714] boot 2024-03-11T22:56:21.286 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:56:21.286 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:21 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: purged_snaps scrub starts 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: purged_snaps scrub ok 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:21.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: Adjusting osd_memory_target on smithi131 to 16064M 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714] boot 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:56:21.453 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:21 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: purged_snaps scrub starts 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: purged_snaps scrub ok 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: Adjusting osd_memory_target on smithi131 to 16064M 2024-03-11T22:56:21.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: osd.1 [v2:172.21.15.131:6800/2065703714,v1:172.21.15.131:6801/2065703714] boot 2024-03-11T22:56:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T22:56:21.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:21 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:56:22.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:22 smithi107 ceph-mon[25849]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:56:22.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:22 smithi107 ceph-mon[25849]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping: /dev/nvme4n1 2024-03-11T22:56:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-03-11T22:56:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-03-11T22:56:23.002 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10+0 records in 2024-03-11T22:56:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:10+0 records out 2024-03-11T22:56:23.002 INFO:teuthology.orchestra.run.smithi195.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0103953 s, 1.0 GB/s 2024-03-11T22:56:23.002 INFO:teuthology.orchestra.run.smithi195.stderr:--> Zapping successful for: 2024-03-11T22:56:23.002 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:22 smithi195 ceph-mon[27640]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:56:23.002 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:22 smithi195 ceph-mon[27640]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:23.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:22 smithi131 ceph-mon[27544]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T22:56:23.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:22 smithi131 ceph-mon[27544]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:23.624 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi195:/dev/nvme4n1 2024-03-11T22:56:23.859 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:56:25.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:25 smithi107 ceph-mon[25849]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:25.559 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:25 smithi131 ceph-mon[27544]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:25.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:25 smithi195 ceph-mon[27640]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:26.397 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:56:26.397 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:56:26.397 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:26.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:56:26.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:56:26.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T22:56:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T22:56:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:27.378 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:27 smithi195 ceph-mon[27640]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:56:27.378 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:27 smithi195 ceph-mon[27640]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:27.379 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:27 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.379 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:27 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.379 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:27 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:27 smithi107 ceph-mon[25849]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:56:27.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:27 smithi107 ceph-mon[25849]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:27.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:27 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:27 smithi131 ceph-mon[27544]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi195:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:56:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:27 smithi131 ceph-mon[27544]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:27 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:27 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:27.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:27 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:29.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:29 smithi107 ceph-mon[25849]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:29 smithi107 ceph-mon[25849]: from='client.? 172.21.15.195:0/949540525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]: dispatch 2024-03-11T22:56:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:29 smithi107 ceph-mon[25849]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]: dispatch 2024-03-11T22:56:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:29 smithi107 ceph-mon[25849]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]': finished 2024-03-11T22:56:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:29 smithi107 ceph-mon[25849]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:56:29.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:29 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:29 smithi131 ceph-mon[27544]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:29 smithi131 ceph-mon[27544]: from='client.? 172.21.15.195:0/949540525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]: dispatch 2024-03-11T22:56:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:29 smithi131 ceph-mon[27544]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]: dispatch 2024-03-11T22:56:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:29 smithi131 ceph-mon[27544]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]': finished 2024-03-11T22:56:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:29 smithi131 ceph-mon[27544]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:56:29.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:29 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:29.571 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:29 smithi195 ceph-mon[27640]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:29.571 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:29 smithi195 ceph-mon[27640]: from='client.? 172.21.15.195:0/949540525' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]: dispatch 2024-03-11T22:56:29.571 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:29 smithi195 ceph-mon[27640]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]: dispatch 2024-03-11T22:56:29.571 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:29 smithi195 ceph-mon[27640]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2f58445-8744-4653-9690-1f2498a90112"}]': finished 2024-03-11T22:56:29.571 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:29 smithi195 ceph-mon[27640]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T22:56:29.571 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:29 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:30.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:30 smithi107 ceph-mon[25849]: from='client.? 172.21.15.195:0/1844812127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:30.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:30 smithi131 ceph-mon[27544]: from='client.? 172.21.15.195:0/1844812127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:30.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:30 smithi195 ceph-mon[27640]: from='client.? 172.21.15.195:0/1844812127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T22:56:31.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:31 smithi107 ceph-mon[25849]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:31.508 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:31 smithi131 ceph-mon[27544]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:31.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:31 smithi195 ceph-mon[27640]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:33.256 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:33 smithi195 ceph-mon[27640]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:33.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:33 smithi107 ceph-mon[25849]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:33.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:33 smithi131 ceph-mon[27544]: pgmap v77: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: Detected new or changed devices on smithi131 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:34 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: Detected new or changed devices on smithi131 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:34.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:34.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:34 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: Detected new or changed devices on smithi131 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:34.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:34.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:34 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:35.559 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:35 smithi131 ceph-mon[27544]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:35.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:35 smithi195 ceph-mon[27640]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:35.603 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:35 smithi107 ceph-mon[25849]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:36.470 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:36 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:56:36.470 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:36 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:36.506 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:36 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:56:36.506 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:36 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:36.615 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:36 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T22:56:36.616 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:36 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:37.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:37 smithi131 ceph-mon[27544]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:37.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:37 smithi131 ceph-mon[27544]: Deploying daemon osd.2 on smithi195 2024-03-11T22:56:37.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:37 smithi195 ceph-mon[27640]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:37.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:37 smithi195 ceph-mon[27640]: Deploying daemon osd.2 on smithi195 2024-03-11T22:56:37.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:37 smithi107 ceph-mon[25849]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:37.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:37 smithi107 ceph-mon[25849]: Deploying daemon osd.2 on smithi195 2024-03-11T22:56:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:39 smithi195 ceph-mon[27640]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:39 smithi131 ceph-mon[27544]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:39 smithi107 ceph-mon[25849]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:41.512 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:41 smithi195 ceph-mon[27640]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:41.559 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:41 smithi131 ceph-mon[27544]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:41.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:41 smithi107 ceph-mon[25849]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:43.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:43 smithi195 ceph-mon[27640]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:43.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:43 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:43 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:43 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:43.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:43 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:43.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:43 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:43.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:43 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:43 smithi107 ceph-mon[25849]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:43.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:43.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:43.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:43.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:43 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:43 smithi131 ceph-mon[27544]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:43.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:43 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:43 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:43.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:43 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:43.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:43 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:43.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:43 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:43.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:43 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:45.528 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:45 smithi195 ceph-mon[27640]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:45.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:45 smithi107 ceph-mon[25849]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:45.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:45 smithi131 ceph-mon[27544]: pgmap v86: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:46.220 INFO:teuthology.orchestra.run.smithi195.stdout:Created osd(s) 2 on host 'smithi195' 2024-03-11T22:56:46.500 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:56:46.500 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:56:46.500 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.500 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:46.500 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.501 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:46.501 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.501 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:46.501 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:46.501 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:46 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:56:46.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:56:46.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:46.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:46.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:46.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:46.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:46 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.809 DEBUG:teuthology.orchestra.run.smithi195:osd.2> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:46 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:46.811 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-03-11T22:56:46.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T22:56:47.052 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: Adjusting osd_memory_target on smithi195 to 16064M 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: from='osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:47 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:47.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: Adjusting osd_memory_target on smithi195 to 16064M 2024-03-11T22:56:47.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:56:47.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: from='osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-03-11T22:56:47.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:56:47.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:47.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-03-11T22:56:47.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: Adjusting osd_memory_target on smithi195 to 16064M 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: from='osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]: dispatch 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:47.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:47 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:48.078 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2[34314]: 2024-03-11T22:56:47.590+0000 7f4f9e814700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T22:56:48.079 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:47 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2[34314]: 2024-03-11T22:56:47.593+0000 7f4f965ff700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T22:56:48.380 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:48 smithi107 ceph-mon[25849]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-03-11T22:56:48.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:48 smithi107 ceph-mon[25849]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:56:48.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:48 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:48.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:48 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:48.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:48 smithi107 ceph-mon[25849]: from='osd.2 ' entity='osd.2' 2024-03-11T22:56:48.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:48 smithi195 ceph-mon[27640]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-03-11T22:56:48.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:48 smithi195 ceph-mon[27640]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:56:48.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:48 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:48.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:48 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:48.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:48 smithi195 ceph-mon[27640]: from='osd.2 ' entity='osd.2' 2024-03-11T22:56:48.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:48 smithi131 ceph-mon[27544]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi195", "root=default"]}]': finished 2024-03-11T22:56:48.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:48 smithi131 ceph-mon[27544]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T22:56:48.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:48 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:48.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:48 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:48.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:48 smithi131 ceph-mon[27544]: from='osd.2 ' entity='osd.2' 2024-03-11T22:56:48.830 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:56:49.689 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: purged_snaps scrub starts 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: purged_snaps scrub ok 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573] boot 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3478459694' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:56:49.690 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:49 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:56:49.714 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1710197808,"num_in_osds":3,"osd_in_since":1710197788,"num_remapped_pgs":0} 2024-03-11T22:56:49.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: purged_snaps scrub starts 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: purged_snaps scrub ok 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573] boot 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3478459694' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:56:49.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:49 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:56:49.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: purged_snaps scrub starts 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: purged_snaps scrub ok 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: osd.2 [v2:172.21.15.195:6800/2571871573,v1:172.21.15.195:6801/2571871573] boot 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3478459694' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T22:56:49.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:49 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:56:50.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:50 smithi107 ceph-mon[25849]: pgmap v92: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:50.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:50 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:56:50.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:50 smithi107 ceph-mon[25849]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:56:50.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:50 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:56:51.018 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:56:51.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:50 smithi131 ceph-mon[27544]: pgmap v92: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:51.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:50 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:56:51.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:50 smithi131 ceph-mon[27544]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:56:51.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:50 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:56:51.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:50 smithi195 ceph-mon[27640]: pgmap v92: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:51.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:50 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T22:56:51.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:50 smithi195 ceph-mon[27640]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T22:56:51.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:50 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T22:56:51.139 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:50 smithi107 sudo[40542]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:56:51.139 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:50 smithi107 sudo[40542]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:51.140 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:50 smithi107 sudo[40542]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:51.140 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:50 smithi107 sudo[40542]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:51.425 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:51 smithi107 sudo[40565]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:56:51.425 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:51 smithi107 sudo[40565]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:51.425 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:51 smithi107 sudo[40565]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:51.425 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:56:51 smithi107 sudo[40565]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:51.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:56:51.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:51 smithi107 ceph-mon[25849]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:56:51.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:51 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:51.728 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:56:51.728 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:51 smithi131 ceph-mon[27544]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:56:51.728 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:51 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:51.728 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34832]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:56:51.729 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34832]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:51.729 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34832]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:51.729 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34832]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:51.990 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T22:56:51.990 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:51 smithi195 ceph-mon[27640]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T22:56:51.990 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:51 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:52.059 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34835]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:56:52.060 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34835]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:52.060 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34835]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:52.060 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:56:51 smithi131 sudo[34835]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:52.244 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:51 smithi195 sudo[35188]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme4n1 2024-03-11T22:56:52.244 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:51 smithi195 sudo[35188]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:52.245 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:51 smithi195 sudo[35188]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:52.245 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:52 smithi195 sudo[35188]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:52.578 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:52 smithi195 sudo[35191]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-03-11T22:56:52.579 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:52 smithi195 sudo[35191]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:52.579 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:52 smithi195 sudo[35191]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:52.579 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:56:52 smithi195 sudo[35191]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:52.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40710]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:56:52.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40710]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:52.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40710]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:52.732 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40710]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:53.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40746]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:56:53.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40746]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:53.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40746]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:53.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 sudo[40746]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:53.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 ceph-mon[25849]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:56:53.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.002 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:52 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.002 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:56:53.002 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":21,"fsid":"eb9eeed0-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:53:21.743186+0000","modified":"2024-03-11T22:56:51.609547+0000","last_up_change":"2024-03-11T22:56:48.593734+0000","last_in_change":"2024-03-11T22:56:28.204174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:56:49.529079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cec53b73-16f2-486e-803c-c603a1502b96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6803","nonce":2877550137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6805","nonce":2877550137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6809","nonce":2877550137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6807","nonce":2877550137}]},"public_addr":"172.21.15.107:6803/2877550137","cluster_addr":"172.21.15.107:6805/2877550137","heartbeat_back_addr":"172.21.15.107:6809/2877550137","heartbeat_front_addr":"172.21.15.107:6807/2877550137","state":["exists","up"]},{"osd":1,"uuid":"c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6801","nonce":2065703714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6803","nonce":2065703714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6807","nonce":2065703714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6805","nonce":2065703714}]},"public_addr":"172.21.15.131:6801/2065703714","cluster_addr":"172.21.15.131:6803/2065703714","heartbeat_back_addr":"172.21.15.131:6807/2065703714","heartbeat_front_addr":"172.21.15.131:6805/2065703714","state":["exists","up"]},{"osd":2,"uuid":"c2f58445-8744-4653-9690-1f2498a90112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6801","nonce":2571871573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6803","nonce":2571871573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6807","nonce":2571871573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6805","nonce":2571871573}]},"public_addr":"172.21.15.195:6801/2571871573","cluster_addr":"172.21.15.195:6803/2571871573","heartbeat_back_addr":"172.21.15.195:6807/2571871573","heartbeat_front_addr":"172.21.15.195:6805/2571871573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:55:54.453440+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:19.043220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:47.149337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/799670939":"2024-03-12T22:54:17.198158+0000","172.21.15.107:6800/799670939":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/4176258396":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/3641577367":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/1783153760":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/2894490165":"2024-03-12T22:53:50.198969+0000","172.21.15.107:6801/3343609294":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/3983068055":"2024-03-12T22:53:50.198969+0000","172.21.15.107:6800/3343609294":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/1676473374":"2024-03-12T22:54:17.198158+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:56:53.056 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:52 smithi195 ceph-mon[27640]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:56:53.056 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:52 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.056 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:52 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:52 smithi131 ceph-mon[27544]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T22:56:53.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:52 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:52 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.308 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35194]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:56:53.308 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35194]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:53.308 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35194]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:53.308 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35194]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:53.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35197]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:56:53.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35197]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:53.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35197]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:53.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 sudo[35197]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:53.671 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T22:56:49.529079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T22:56:53.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T22:56:53.821 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34906]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T22:56:53.821 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34906]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:53.822 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34906]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:53.822 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34906]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:53.911 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:53.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3058525701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:53.953 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:53 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:54.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:54.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3058525701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:54.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:53 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34909]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34909]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34909]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 sudo[34909]: pam_unix(sudo:session): session closed for user root 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: pgmap v95: 1 pgs: 1 active+clean; 449 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:54.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: mgrmap e16: y(active, since 2m), standbys: z, x 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3058525701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:54.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:56:54.312 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:56:54.312 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:53 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:56:55.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: pgmap v97: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: Detected new or changed devices on smithi195 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: pgmap v98: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:55.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.080 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:54 smithi195 ceph-mon[27640]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-11T22:56:55.135 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:55.135 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: pgmap v97: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: Detected new or changed devices on smithi195 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: pgmap v98: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.136 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:54 smithi107 ceph-mon[25849]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: pgmap v97: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: Detected new or changed devices on smithi195 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:55.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:55.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:55.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: pgmap v98: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:55.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:55.311 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:54 smithi131 ceph-mon[27544]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-03-11T22:56:55.682 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 1 2024-03-11T22:56:56.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: pgmap v99: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:55 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3897006952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: pgmap v99: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.232 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:55 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3897006952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:56:56.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:56:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:56:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:56:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: pgmap v99: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:56:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:56:56.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:55 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3897006952' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T22:56:56.326 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi107 2024-03-11T22:56:56.326 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi107=foo.a' 2024-03-11T22:56:56.703 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:56:57.224 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:56 smithi195 ceph-mon[27640]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-11T22:56:57.225 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:56 smithi195 ceph-mon[27640]: Cluster is now healthy 2024-03-11T22:56:57.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:56 smithi131 ceph-mon[27544]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-11T22:56:57.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:56 smithi131 ceph-mon[27544]: Cluster is now healthy 2024-03-11T22:56:57.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:56 smithi107 ceph-mon[25849]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-03-11T22:56:57.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:56 smithi107 ceph-mon[25849]: Cluster is now healthy 2024-03-11T22:56:58.238 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:56:57 smithi195 ceph-mon[27640]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 609 KiB/s wr, 39 op/s 2024-03-11T22:56:58.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:56:57 smithi131 ceph-mon[27544]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 609 KiB/s wr, 39 op/s 2024-03-11T22:56:58.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:56:57 smithi107 ceph-mon[25849]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 11 KiB/s rd, 609 KiB/s wr, 39 op/s 2024-03-11T22:56:59.199 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled rgw.foo.a update... 2024-03-11T22:56:59.808 DEBUG:teuthology.orchestra.run.smithi107:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T22:56:59.810 INFO:tasks.cephadm:Adding prometheus.a on smithi131 2024-03-11T22:56:59.811 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi131=a' 2024-03-11T22:57:00.043 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:57:00.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 8.3 KiB/s rd, 448 KiB/s wr, 28 op/s 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi107=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:00.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:00 smithi107 ceph-mon[25849]: Deploying daemon rgw.foo.a on smithi107 2024-03-11T22:57:00.456 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 8.3 KiB/s rd, 448 KiB/s wr, 28 op/s 2024-03-11T22:57:00.456 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi107=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:00.456 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-03-11T22:57:00.456 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:00.457 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:00 smithi195 ceph-mon[27640]: Deploying daemon rgw.foo.a on smithi107 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 8.3 KiB/s rd, 448 KiB/s wr, 28 op/s 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi107=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:00.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T22:57:00.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T22:57:00.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:00.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:00.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:00 smithi131 ceph-mon[27544]: Deploying daemon rgw.foo.a on smithi107 2024-03-11T22:57:01.452 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:57:01 smithi107 systemd[1]: Starting Ceph rgw.foo.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:57:01.723 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled prometheus update... 2024-03-11T22:57:01.758 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:57:01 smithi107 podman[41355]: 2024-03-11 22:57:01.352261384 +0000 UTC m=+0.017299331 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:57:01.759 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:57:01 smithi107 podman[41355]: 2024-03-11 22:57:01.469487702 +0000 UTC m=+0.134525690 container create 998ec625860a8c61e7e2793e1a5daeaecb756b45c9e32b41801700368b56efae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T22:57:01.759 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:57:01 smithi107 podman[41355]: 2024-03-11 22:57:01.688304698 +0000 UTC m=+0.353342647 container init 998ec625860a8c61e7e2793e1a5daeaecb756b45c9e32b41801700368b56efae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T22:57:01.759 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:57:01 smithi107 podman[41355]: 2024-03-11 22:57:01.693269773 +0000 UTC m=+0.358307716 container start 998ec625860a8c61e7e2793e1a5daeaecb756b45c9e32b41801700368b56efae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T22:57:01.759 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:57:01 smithi107 bash[41355]: 998ec625860a8c61e7e2793e1a5daeaecb756b45c9e32b41801700368b56efae 2024-03-11T22:57:02.200 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:57:01 smithi107 systemd[1]: Started Ceph rgw.foo.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:57:02.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 363 KiB/s wr, 23 op/s 2024-03-11T22:57:02.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.216 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.216 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.216 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.216 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.216 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:02.216 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:02.216 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:02 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:02.241 DEBUG:teuthology.orchestra.run.smithi131:prometheus.a> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T22:57:02.243 INFO:tasks.cephadm:Adding node-exporter.a on smithi107 2024-03-11T22:57:02.243 INFO:tasks.cephadm:Adding node-exporter.b on smithi131 2024-03-11T22:57:02.243 INFO:tasks.cephadm:Adding node-exporter.c on smithi195 2024-03-11T22:57:02.243 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '3;smithi107=a;smithi131=b;smithi195=c' 2024-03-11T22:57:02.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 363 KiB/s wr, 23 op/s 2024-03-11T22:57:02.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:02.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:02 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:02.474 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 6.7 KiB/s rd, 363 KiB/s wr, 23 op/s 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:02.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:02 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:03.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi131=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: Saving service prometheus spec with placement smithi131=a;count:1 2024-03-11T22:57:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-03-11T22:57:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: Checking dashboard <-> RGW credentials 2024-03-11T22:57:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:57:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3233545250' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:57:03.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:03 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1470982141' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi131=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: Saving service prometheus spec with placement smithi131=a;count:1 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: Checking dashboard <-> RGW credentials 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3233545250' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:57:03.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:03 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1470982141' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi131=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: Saving service prometheus spec with placement smithi131=a;count:1 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: Saving service rgw.foo.a spec with placement smithi107=foo.a;count:1 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: Checking dashboard <-> RGW credentials 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3233545250' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:57:03.572 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:03 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1470982141' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T22:57:04.350 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled node-exporter update... 2024-03-11T22:57:04.402 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:04 smithi107 ceph-mon[25849]: pgmap v104: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 385 KiB/s wr, 24 op/s 2024-03-11T22:57:04.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:04 smithi107 ceph-mon[25849]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:57:04.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:04 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3233545250' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:57:04.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:04 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1470982141' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:57:04.403 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:04 smithi107 ceph-mon[25849]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:57:04.661 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:04 smithi195 ceph-mon[27640]: pgmap v104: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 385 KiB/s wr, 24 op/s 2024-03-11T22:57:04.661 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:04 smithi195 ceph-mon[27640]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:57:04.661 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:04 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3233545250' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:57:04.661 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:04 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1470982141' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:57:04.661 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:04 smithi195 ceph-mon[27640]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:57:04.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:04 smithi131 ceph-mon[27544]: pgmap v104: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail; 7.1 KiB/s rd, 385 KiB/s wr, 24 op/s 2024-03-11T22:57:04.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:04 smithi131 ceph-mon[27544]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T22:57:04.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:04 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3233545250' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:57:04.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:04 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1470982141' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T22:57:04.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:04 smithi131 ceph-mon[27544]: osdmap e23: 3 total, 3 up, 3 in 2024-03-11T22:57:04.971 DEBUG:teuthology.orchestra.run.smithi107:node-exporter.a> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T22:57:04.973 DEBUG:teuthology.orchestra.run.smithi131:node-exporter.b> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T22:57:04.976 DEBUG:teuthology.orchestra.run.smithi195:node-exporter.c> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.c.service 2024-03-11T22:57:04.978 INFO:tasks.cephadm:Adding alertmanager.a on smithi107 2024-03-11T22:57:04.979 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi107=a' 2024-03-11T22:57:05.247 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:57:05.672 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:05 smithi195 ceph-mon[27640]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi107=a;smithi131=b;smithi195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:05.673 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:05 smithi195 ceph-mon[27640]: Saving service node-exporter spec with placement smithi107=a;smithi131=b;smithi195=c;count:3 2024-03-11T22:57:05.673 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:05 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:05.673 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:05 smithi195 ceph-mon[27640]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:57:05.673 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:05 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:57:05.673 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:05 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:57:05.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:05 smithi107 ceph-mon[25849]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi107=a;smithi131=b;smithi195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:05.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:05 smithi107 ceph-mon[25849]: Saving service node-exporter spec with placement smithi107=a;smithi131=b;smithi195=c;count:3 2024-03-11T22:57:05.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:05 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:05.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:05 smithi107 ceph-mon[25849]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:57:05.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:05 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:57:05.703 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:05 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:57:05.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:05 smithi131 ceph-mon[27544]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi107=a;smithi131=b;smithi195=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:05.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:05 smithi131 ceph-mon[27544]: Saving service node-exporter spec with placement smithi107=a;smithi131=b;smithi195=c;count:3 2024-03-11T22:57:05.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:05 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:05.811 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:05 smithi131 ceph-mon[27544]: osdmap e24: 3 total, 3 up, 3 in 2024-03-11T22:57:05.811 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:05 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:57:05.811 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:05 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T22:57:06.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:06 smithi107 ceph-mon[25849]: pgmap v107: 65 pgs: 50 unknown, 15 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-11T22:57:06.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:06 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:57:06.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:06 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:57:06.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:06 smithi107 ceph-mon[25849]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:57:06.603 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:06 smithi195 ceph-mon[27640]: pgmap v107: 65 pgs: 50 unknown, 15 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-11T22:57:06.603 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:06 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:57:06.604 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:06 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:57:06.604 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:06 smithi195 ceph-mon[27640]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:57:06.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:06 smithi131 ceph-mon[27544]: pgmap v107: 65 pgs: 50 unknown, 15 active+clean; 577 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-11T22:57:06.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:06 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:57:06.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:06 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T22:57:06.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:06 smithi131 ceph-mon[27544]: osdmap e25: 3 total, 3 up, 3 in 2024-03-11T22:57:07.035 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled alertmanager update... 2024-03-11T22:57:07.667 DEBUG:teuthology.orchestra.run.smithi107:alertmanager.a> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T22:57:07.670 INFO:tasks.cephadm:Adding grafana.a on smithi131 2024-03-11T22:57:07.670 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi131=a' 2024-03-11T22:57:07.918 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:57:07.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:07.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: pgmap v110: 97 pgs: 61 unknown, 36 active+clean; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: Saving service alertmanager spec with placement smithi107=a;count:1 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:07.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:07 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.039 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.039 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: pgmap v110: 97 pgs: 61 unknown, 36 active+clean; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: Saving service alertmanager spec with placement smithi107=a;count:1 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.040 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:07 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: osdmap e26: 3 total, 3 up, 3 in 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: pgmap v110: 97 pgs: 61 unknown, 36 active+clean; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi107=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: Saving service alertmanager spec with placement smithi107=a;count:1 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:08.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:07 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:09.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:08 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:57:09.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:08 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:57:09.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:08 smithi131 ceph-mon[27544]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:57:09.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:08 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:57:09.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:08 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:57:09.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:08 smithi195 ceph-mon[27640]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:57:09.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:08 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:57:09.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:08 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T22:57:09.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:08 smithi107 ceph-mon[25849]: osdmap e27: 3 total, 3 up, 3 in 2024-03-11T22:57:09.666 INFO:teuthology.orchestra.run.smithi195.stdout:Scheduled grafana update... 2024-03-11T22:57:10.009 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:09 smithi195 ceph-mon[27640]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:57:10.009 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:09 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:57:10.010 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:09 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:57:10.010 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:09 smithi195 ceph-mon[27640]: pgmap v113: 129 pgs: 11 creating+peering, 56 active+clean, 62 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:57:10.010 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:09 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:10.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:09 smithi131 ceph-mon[27544]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:57:10.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:09 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:57:10.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:09 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:57:10.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:09 smithi131 ceph-mon[27544]: pgmap v113: 129 pgs: 11 creating+peering, 56 active+clean, 62 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:57:10.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:09 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:10.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:09 smithi107 ceph-mon[25849]: osdmap e28: 3 total, 3 up, 3 in 2024-03-11T22:57:10.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:09 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:57:10.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:09 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T22:57:10.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:09 smithi107 ceph-mon[25849]: pgmap v113: 129 pgs: 11 creating+peering, 56 active+clean, 62 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-11T22:57:10.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:09 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:10.312 DEBUG:teuthology.orchestra.run.smithi131:grafana.a> sudo journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T22:57:10.314 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T22:57:10.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:57:10.560 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:10 smithi107 ceph-mon[25849]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi131=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:10 smithi107 ceph-mon[25849]: Saving service grafana spec with placement smithi131=a;count:1 2024-03-11T22:57:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:10 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:57:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:10 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:57:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:10 smithi107 ceph-mon[25849]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:57:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:10 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:57:10.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:10 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:57:11.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:10 smithi131 ceph-mon[27544]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi131=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:11.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:10 smithi131 ceph-mon[27544]: Saving service grafana spec with placement smithi131=a;count:1 2024-03-11T22:57:11.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:10 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:57:11.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:10 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:57:11.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:10 smithi131 ceph-mon[27544]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:57:11.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:10 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:57:11.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:10 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:57:11.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:10 smithi195 ceph-mon[27640]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi131=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:57:11.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:10 smithi195 ceph-mon[27640]: Saving service grafana spec with placement smithi131=a;count:1 2024-03-11T22:57:11.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:10 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:57:11.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:10 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T22:57:11.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:10 smithi195 ceph-mon[27640]: osdmap e29: 3 total, 3 up, 3 in 2024-03-11T22:57:11.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:10 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:57:11.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:10 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T22:57:11.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:57:11.981 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:57:11.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:57:11.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:11.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:11.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:11.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: pgmap v116: 129 pgs: 20 creating+peering, 77 active+clean, 32 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-11T22:57:11.982 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:11 smithi107 ceph-mon[25849]: Deploying daemon prometheus.a on smithi131 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: pgmap v116: 129 pgs: 20 creating+peering, 77 active+clean, 32 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-11T22:57:12.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:11 smithi131 ceph-mon[27544]: Deploying daemon prometheus.a on smithi131 2024-03-11T22:57:12.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1813635788' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:57:12.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/85201885' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T22:57:12.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: osdmap e30: 3 total, 3 up, 3 in 2024-03-11T22:57:12.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:12.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:12.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:12.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: pgmap v116: 129 pgs: 20 creating+peering, 77 active+clean, 32 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 511 B/s wr, 0 op/s 2024-03-11T22:57:12.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:11 smithi195 ceph-mon[27640]: Deploying daemon prometheus.a on smithi131 2024-03-11T22:57:12.668 INFO:teuthology.orchestra.run.smithi107.stdout:[client.0] 2024-03-11T22:57:12.668 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBIjO9lvduEJxAAvfg+BZ20aXIT0jvzwBI5/w== 2024-03-11T22:57:12.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:12 smithi107 ceph-mon[25849]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:57:12.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:12 smithi107 ceph-mon[25849]: Cluster is now healthy 2024-03-11T22:57:12.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:12 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/423083650' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:12.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:12 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/423083650' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:13.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:12 smithi131 ceph-mon[27544]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:57:13.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:12 smithi131 ceph-mon[27544]: Cluster is now healthy 2024-03-11T22:57:13.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:12 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/423083650' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:13.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:12 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/423083650' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:13.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:12 smithi195 ceph-mon[27640]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T22:57:13.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:12 smithi195 ceph-mon[27640]: Cluster is now healthy 2024-03-11T22:57:13.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:12 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/423083650' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:13.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:12 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/423083650' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:13.354 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:57:13.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T22:57:13.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T22:57:13.396 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:57:13.653 INFO:teuthology.orchestra.run.smithi131.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T22:57:14.032 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:13 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.032 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:13 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.032 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:13 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.032 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:13 smithi131 ceph-mon[27544]: pgmap v117: 129 pgs: 9 creating+peering, 120 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 64 KiB/s rd, 4.6 KiB/s wr, 155 op/s 2024-03-11T22:57:14.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:13 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:13 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:13 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:13 smithi195 ceph-mon[27640]: pgmap v117: 129 pgs: 9 creating+peering, 120 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 64 KiB/s rd, 4.6 KiB/s wr, 155 op/s 2024-03-11T22:57:14.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:13 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:13 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:13 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:14.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:13 smithi107 ceph-mon[25849]: pgmap v117: 129 pgs: 9 creating+peering, 120 active+clean; 581 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 64 KiB/s rd, 4.6 KiB/s wr, 155 op/s 2024-03-11T22:57:15.429 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:15 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.429 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:15 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.429 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:15 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:15 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:15 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:15 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:15 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:15 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:15.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:15 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:16.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:16 smithi131 ceph-mon[27544]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-03-11T22:57:16.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:16 smithi195 ceph-mon[27640]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-03-11T22:57:16.329 INFO:teuthology.orchestra.run.smithi131.stdout:[client.1] 2024-03-11T22:57:16.330 INFO:teuthology.orchestra.run.smithi131.stdout: key = AQBMjO9lgVcHExAAed9f56WpHAyW7r0YlL0hNg== 2024-03-11T22:57:16.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:16 smithi107 ceph-mon[25849]: pgmap v118: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 75 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-03-11T22:57:17.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:17 smithi195 ceph-mon[27640]: from='client.? 172.21.15.131:0/3937099193' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:17.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:17 smithi195 ceph-mon[27640]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:17.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:17 smithi195 ceph-mon[27640]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:17.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:17 smithi107 ceph-mon[25849]: from='client.? 172.21.15.131:0/3937099193' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:17.377 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:17 smithi107 ceph-mon[25849]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:17.378 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:17 smithi107 ceph-mon[25849]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:17.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:17 smithi131 ceph-mon[27544]: from='client.? 172.21.15.131:0/3937099193' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:17.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:17 smithi131 ceph-mon[27544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:17.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:17 smithi131 ceph-mon[27544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:18.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:18 smithi195 ceph-mon[27640]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:57:18.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:18 smithi107 ceph-mon[25849]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:57:18.772 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:18 smithi131 ceph-mon[27544]: pgmap v119: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-03-11T22:57:19.002 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:57:19.002 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T22:57:19.002 DEBUG:teuthology.orchestra.run.smithi131:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T22:57:19.046 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T22:57:19.294 INFO:teuthology.orchestra.run.smithi195.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.c/config 2024-03-11T22:57:20.538 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:20 smithi195 ceph-mon[27640]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 50 KiB/s rd, 3.9 KiB/s wr, 122 op/s 2024-03-11T22:57:20.544 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:20 smithi131 ceph-mon[27544]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 50 KiB/s rd, 3.9 KiB/s wr, 122 op/s 2024-03-11T22:57:20.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:20 smithi107 ceph-mon[25849]: pgmap v120: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 50 KiB/s rd, 3.9 KiB/s wr, 122 op/s 2024-03-11T22:57:21.151 INFO:teuthology.orchestra.run.smithi195.stdout:[client.2] 2024-03-11T22:57:21.151 INFO:teuthology.orchestra.run.smithi195.stdout: key = AQBRjO9ltuaNCBAA+PFbEoPeBZzrfM2LVqg2Dg== 2024-03-11T22:57:21.502 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:21 smithi195 ceph-mon[27640]: from='client.? 172.21.15.195:0/885182609' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:21.502 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:21 smithi195 ceph-mon[27640]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:21.502 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:21 smithi195 ceph-mon[27640]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:21.519 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:21 smithi131 ceph-mon[27544]: from='client.? 172.21.15.195:0/885182609' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:21.519 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:21 smithi131 ceph-mon[27544]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:21.519 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:21 smithi131 ceph-mon[27544]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:21.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:21 smithi107 ceph-mon[25849]: from='client.? 172.21.15.195:0/885182609' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:21.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:21 smithi107 ceph-mon[25849]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T22:57:21.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:21 smithi107 ceph-mon[25849]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T22:57:21.786 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:57:21.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-03-11T22:57:21.786 DEBUG:teuthology.orchestra.run.smithi195:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-11T22:57:21.828 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T22:57:21.828 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T22:57:21.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T22:57:22.071 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:22.431 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:22 smithi107 ceph-mon[25849]: pgmap v121: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 3.5 KiB/s wr, 110 op/s 2024-03-11T22:57:22.490 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:22 smithi131 ceph-mon[27544]: pgmap v121: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 3.5 KiB/s wr, 110 op/s 2024-03-11T22:57:22.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:22 smithi195 ceph-mon[27640]: pgmap v121: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 45 KiB/s rd, 3.5 KiB/s wr, 110 op/s 2024-03-11T22:57:23.680 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:23 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:23.681 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:23 smithi131 ceph-mon[27544]: pgmap v122: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:57:23.681 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:23 smithi131 systemd[1]: Starting Ceph prometheus.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:57:23.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:23 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:23.746 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:23 smithi107 ceph-mon[25849]: pgmap v122: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:57:23.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:23 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:23.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:23 smithi195 ceph-mon[27640]: pgmap v122: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 39 KiB/s rd, 3.0 KiB/s wr, 93 op/s 2024-03-11T22:57:24.020 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:23 smithi131 podman[36093]: 2024-03-11 22:57:23.79565657 +0000 UTC m=+0.019146583 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T22:57:24.020 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:23 smithi131 podman[36093]: 2024-03-11 22:57:23.910745401 +0000 UTC m=+0.134235399 container create 4116e8e9bc5bfb5015415796c121d0ad19cd9bcdacf273a386d1ee8541536da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:57:24.029 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:57:24.399 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 podman[36093]: 2024-03-11 22:57:24.121214869 +0000 UTC m=+0.344704876 container init 4116e8e9bc5bfb5015415796c121d0ad19cd9bcdacf273a386d1ee8541536da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:57:24.399 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 podman[36093]: 2024-03-11 22:57:24.12619567 +0000 UTC m=+0.349685669 container start 4116e8e9bc5bfb5015415796c121d0ad19cd9bcdacf273a386d1ee8541536da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:57:24.399 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.149Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T22:57:24.399 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.149Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T22:57:24.399 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.150Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi131 (none))" 2024-03-11T22:57:24.399 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.150Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.150Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.151Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.151Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.152Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.152Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.154Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.154Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=17.541µs 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.154Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.154Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.154Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=37.66µs wal_replay_duration=244.005µs wbl_replay_duration=136ns total_replay_duration=402.41µs 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.155Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.155Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.155Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T22:57:24.400 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.173Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.834728ms db_storage=709ns remote_storage=1.069µs web_handler=298ns query_engine=549ns scrape=433.344µs scrape_sd=71.747µs notify=557ns notify_sd=1.498µs rules=17.164309ms tracing=5.592µs 2024-03-11T22:57:24.401 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.173Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T22:57:24.401 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:57:24.173Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T22:57:24.401 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 bash[36093]: 4116e8e9bc5bfb5015415796c121d0ad19cd9bcdacf273a386d1ee8541536da0 2024-03-11T22:57:24.401 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:57:24 smithi131 systemd[1]: Started Ceph prometheus.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:57:24.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1469825194' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:57:24.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.696 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:24 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:57:24.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:24 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1469825194' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:57:24.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.710 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:24 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:57:24.736 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":2813482561},{"type":"v1","addr":"172.21.15.107:6801","nonce":2813482561}]},"active_addr":"172.21.15.107:6801/2813482561","active_change":"2024-03-11T22:54:17.198299+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.107:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3387228046}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":514114490}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3397249869}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":4256541872}]}]} 2024-03-11T22:57:24.741 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T22:57:24.741 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T22:57:24.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:57:24.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:24 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1469825194' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T22:57:24.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' 2024-03-11T22:57:24.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:24 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T22:57:25.216 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:25.524 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:25 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: ignoring --setuser ceph since I am not root 2024-03-11T22:57:25.525 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:25 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: ignoring --setgroup ceph since I am not root 2024-03-11T22:57:25.525 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:25 smithi131 ceph-mgr[29265]: -- 172.21.15.131:0/1445651024 <== mon.2 v2:172.21.15.131:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56358eafa1a0 con 0x56358ead0800 2024-03-11T22:57:25.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:25 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: ignoring --setuser ceph since I am not root 2024-03-11T22:57:25.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:25 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: ignoring --setgroup ceph since I am not root 2024-03-11T22:57:25.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:25 smithi195 ceph-mgr[30117]: -- 172.21.15.195:0/2918227635 <== mon.0 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e6bae721a0 con 0x55e6bae48c00 2024-03-11T22:57:25.670 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:25 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ignoring --setuser ceph since I am not root 2024-03-11T22:57:25.670 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:25 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ignoring --setgroup ceph since I am not root 2024-03-11T22:57:25.925 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:25 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:25.871+0000 7fde05306200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:57:26.063 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:25 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:25.858+0000 7eff5be3d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:57:26.063 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:25 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:25.951+0000 7eff5be3d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:57:26.203 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:25 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:25.966+0000 7fde05306200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:57:26.292 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:25 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:25.855+0000 7f40b49b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T22:57:26.292 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:25 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:25.948+0000 7f40b49b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T22:57:26.559 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:26 smithi131 ceph-mon[27544]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:57:26.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:26 smithi131 ceph-mon[27544]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:57:26.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:26 smithi195 ceph-mon[27640]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:57:26.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:26 smithi195 ceph-mon[27640]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:57:26.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:26 smithi107 ceph-mon[25849]: from='mgr.14152 172.21.15.107:0/1764947662' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T22:57:26.648 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:26 smithi107 ceph-mon[25849]: mgrmap e17: y(active, since 3m), standbys: z, x 2024-03-11T22:57:26.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:26 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:26.642+0000 7fde05306200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:57:26.906 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:26 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:26.751+0000 7fde05306200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:57:26.907 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:26 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:26.901+0000 7fde05306200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:57:26.917 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:26.584+0000 7eff5be3d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:57:26.917 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:26.677+0000 7eff5be3d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:57:26.917 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:26.817+0000 7eff5be3d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:57:26.917 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:26.912+0000 7eff5be3d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:57:26.948 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:26 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:26.597+0000 7f40b49b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T22:57:26.948 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:26 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:26.695+0000 7f40b49b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T22:57:26.948 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:26 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:26.843+0000 7f40b49b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T22:57:27.200 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:27 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:26.999+0000 7fde05306200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:57:27.200 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:27 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:27.083+0000 7fde05306200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:57:27.262 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:26.995+0000 7eff5be3d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:57:27.303 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:26 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:26.943+0000 7f40b49b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T22:57:27.303 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:27 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:27.028+0000 7f40b49b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T22:57:27.303 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:27 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:27.299+0000 7f40b49b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:57:27.451 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:27 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:27.369+0000 7fde05306200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:57:27.556 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:27 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:27.257+0000 7eff5be3d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T22:57:27.556 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:27 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:27.369+0000 7eff5be3d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:57:27.556 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:27 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:27.456+0000 7eff5be3d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:57:27.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:27 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:27.410+0000 7f40b49b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:57:27.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:27 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:27.499+0000 7f40b49b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:57:27.809 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:27 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:27.551+0000 7eff5be3d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:57:27.810 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:27 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:27.727+0000 7eff5be3d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:57:27.815 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:27 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:27.493+0000 7fde05306200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T22:57:27.815 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:27 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:27.580+0000 7fde05306200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T22:57:27.815 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:27 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:27.677+0000 7fde05306200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:57:28.070 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:27 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:27.858+0000 7fde05306200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:57:28.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:27 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:27.592+0000 7f40b49b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T22:57:28.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:27 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:27.767+0000 7f40b49b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T22:57:28.409 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:57:28.409 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"fsid":"eb9eeed0-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:53:21.743186+0000","modified":"2024-03-11T22:57:10.748070+0000","last_up_change":"2024-03-11T22:56:48.593734+0000","last_in_change":"2024-03-11T22:56:28.204174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:56:49.529079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T22:57:01.966268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:57:04.049221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:57:05.986054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:57:08.032101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cec53b73-16f2-486e-803c-c603a1502b96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6803","nonce":2877550137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6805","nonce":2877550137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6809","nonce":2877550137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6807","nonce":2877550137}]},"public_addr":"172.21.15.107:6803/2877550137","cluster_addr":"172.21.15.107:6805/2877550137","heartbeat_back_addr":"172.21.15.107:6809/2877550137","heartbeat_front_addr":"172.21.15.107:6807/2877550137","state":["exists","up"]},{"osd":1,"uuid":"c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6801","nonce":2065703714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6803","nonce":2065703714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6807","nonce":2065703714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6805","nonce":2065703714}]},"public_addr":"172.21.15.131:6801/2065703714","cluster_addr":"172.21.15.131:6803/2065703714","heartbeat_back_addr":"172.21.15.131:6807/2065703714","heartbeat_front_addr":"172.21.15.131:6805/2065703714","state":["exists","up"]},{"osd":2,"uuid":"c2f58445-8744-4653-9690-1f2498a90112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6801","nonce":2571871573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6803","nonce":2571871573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6807","nonce":2571871573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6805","nonce":2571871573}]},"public_addr":"172.21.15.195:6801/2571871573","cluster_addr":"172.21.15.195:6803/2571871573","heartbeat_back_addr":"172.21.15.195:6807/2571871573","heartbeat_front_addr":"172.21.15.195:6805/2571871573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:55:54.453440+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:19.043220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:47.149337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/799670939":"2024-03-12T22:54:17.198158+0000","172.21.15.107:6800/799670939":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/4176258396":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/3641577367":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/1783153760":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/2894490165":"2024-03-12T22:53:50.198969+0000","172.21.15.107:6801/3343609294":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/3983068055":"2024-03-12T22:53:50.198969+0000","172.21.15.107:6800/3343609294":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/1676473374":"2024-03-12T22:54:17.198158+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:57:28.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:28 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/301974665' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:57:28.724 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:28 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/301974665' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:57:28.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:28 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/301974665' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:57:28.810 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:28 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:28.686+0000 7eff5be3d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:57:28.967 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:28 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:28.854+0000 7fde05306200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:57:28.996 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T22:57:28.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T22:57:29.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:28 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:28.719+0000 7f40b49b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T22:57:29.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:28 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:28.849+0000 7f40b49b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:57:29.156 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:28 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:28.817+0000 7eff5be3d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:57:29.240 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:29.356 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:28 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:28.992+0000 7fde05306200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T22:57:29.356 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:29 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:29.351+0000 7fde05306200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:57:29.445 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:29 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:29.152+0000 7eff5be3d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:57:29.445 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:29 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:29.241+0000 7eff5be3d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:57:29.482 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:29.195+0000 7f40b49b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T22:57:29.483 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:29.285+0000 7f40b49b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:57:29.483 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:29.478+0000 7f40b49b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:57:29.653 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:29 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:29.445+0000 7fde05306200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T22:57:29.809 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:29 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:29.440+0000 7eff5be3d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:57:29.810 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:29 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:29.622+0000 7eff5be3d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:57:29.828 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:29.660+0000 7f40b49b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:57:29.951 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:29 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:29.649+0000 7fde05306200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:57:29.951 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:29 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:29.842+0000 7fde05306200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T22:57:30.168 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:29 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:29.904+0000 7eff5be3d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:57:30.207 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:29 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:29.942+0000 7f40b49b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:57:30.331 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:30 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:30.136+0000 7fde05306200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T22:57:30.419 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:30 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:30.164+0000 7eff5be3d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:57:30.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:30 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:30.203+0000 7f40b49b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:57:30.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:30 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:30.435+0000 7f40b49b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:57:30.608 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:30 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:30.403+0000 7fde05306200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T22:57:30.809 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:30 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:30.414+0000 7eff5be3d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:57:30.859 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:30 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:30.652+0000 7fde05306200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T22:57:31.067 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:57:31.067 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"fsid":"eb9eeed0-dff9-11ee-95c8-87774f69a715","created":"2024-03-11T22:53:21.743186+0000","modified":"2024-03-11T22:57:10.748070+0000","last_up_change":"2024-03-11T22:56:48.593734+0000","last_in_change":"2024-03-11T22:56:28.204174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T22:56:49.529079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T22:57:01.966268+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-03-11T22:57:04.049221+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-03-11T22:57:05.986054+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-03-11T22:57:08.032101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cec53b73-16f2-486e-803c-c603a1502b96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6803","nonce":2877550137}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6805","nonce":2877550137}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6809","nonce":2877550137}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":2877550137},{"type":"v1","addr":"172.21.15.107:6807","nonce":2877550137}]},"public_addr":"172.21.15.107:6803/2877550137","cluster_addr":"172.21.15.107:6805/2877550137","heartbeat_back_addr":"172.21.15.107:6809/2877550137","heartbeat_front_addr":"172.21.15.107:6807/2877550137","state":["exists","up"]},{"osd":1,"uuid":"c0c4a9bd-a8c9-45dc-a7d5-e66748ac8a18","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6800","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6801","nonce":2065703714}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6802","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6803","nonce":2065703714}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6806","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6807","nonce":2065703714}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.131:6804","nonce":2065703714},{"type":"v1","addr":"172.21.15.131:6805","nonce":2065703714}]},"public_addr":"172.21.15.131:6801/2065703714","cluster_addr":"172.21.15.131:6803/2065703714","heartbeat_back_addr":"172.21.15.131:6807/2065703714","heartbeat_front_addr":"172.21.15.131:6805/2065703714","state":["exists","up"]},{"osd":2,"uuid":"c2f58445-8744-4653-9690-1f2498a90112","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6800","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6801","nonce":2571871573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6802","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6803","nonce":2571871573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6806","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6807","nonce":2571871573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.195:6804","nonce":2571871573},{"type":"v1","addr":"172.21.15.195:6805","nonce":2571871573}]},"public_addr":"172.21.15.195:6801/2571871573","cluster_addr":"172.21.15.195:6803/2571871573","heartbeat_back_addr":"172.21.15.195:6807/2571871573","heartbeat_front_addr":"172.21.15.195:6805/2571871573","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:55:54.453440+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:19.043220+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T22:56:47.149337+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.107:6801/799670939":"2024-03-12T22:54:17.198158+0000","172.21.15.107:6800/799670939":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/4176258396":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/3641577367":"2024-03-12T22:54:17.198158+0000","172.21.15.107:0/1783153760":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/2894490165":"2024-03-12T22:53:50.198969+0000","172.21.15.107:6801/3343609294":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/3983068055":"2024-03-12T22:53:50.198969+0000","172.21.15.107:6800/3343609294":"2024-03-12T22:53:50.198969+0000","172.21.15.107:0/1676473374":"2024-03-12T22:54:17.198158+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T22:57:31.371 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:31 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/1698293284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:57:31.489 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:31 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/1698293284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:57:31.489 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:31 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:31.248+0000 7eff5be3d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:57:31.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:31 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/1698293284' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T22:57:31.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:31 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:31.232+0000 7f40b49b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:57:31.637 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:31 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:31.490+0000 7fde05306200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T22:57:31.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T22:57:31.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T22:57:31.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T22:57:31.928 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:31.987 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:32.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:31 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:31.800+0000 7f40b49b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:57:32.078 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:31 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:31.898+0000 7f40b49b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:57:32.190 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:31 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:31.868+0000 7eff5be3d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:57:32.190 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:31 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:31.976+0000 7eff5be3d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:32.112+0000 7fde05306200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:32.217+0000 7fde05306200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-mon[25849]: Standby manager daemon z restarted 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-mon[25849]: Standby manager daemon z started 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:57:32.338 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:57:32.559 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-mon[27544]: Standby manager daemon z restarted 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-mon[27544]: Standby manager daemon z started 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:32.202+0000 7eff5be3d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: [11/Mar/2024:22:57:32] ENGINE Bus STARTING 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: CherryPy Checker: 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: The Application mounted at '' has an empty config. 2024-03-11T22:57:32.560 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: 2024-03-11T22:57:32.561 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: [11/Mar/2024:22:57:32] ENGINE Serving on http://:::9283 2024-03-11T22:57:32.561 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:57:32 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x[29242]: [11/Mar/2024:22:57:32] ENGINE Bus STARTED 2024-03-11T22:57:32.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-mon[27640]: Standby manager daemon z restarted 2024-03-11T22:57:32.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-mon[27640]: Standby manager daemon z started 2024-03-11T22:57:32.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-11T22:57:32.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:57:32.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-11T22:57:32.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.195:0/2348031932' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:57:32.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:32.110+0000 7f40b49b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:57:32.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: [11/Mar/2024:22:57:32] ENGINE Bus STARTING 2024-03-11T22:57:32.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: CherryPy Checker: 2024-03-11T22:57:32.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: The Application mounted at '' has an empty config. 2024-03-11T22:57:32.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: 2024-03-11T22:57:32.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: [11/Mar/2024:22:57:32] ENGINE Serving on http://:::9283 2024-03-11T22:57:32.579 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:57:32 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z[30062]: [11/Mar/2024:22:57:32] ENGINE Bus STARTED 2024-03-11T22:57:32.650 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:32.453+0000 7fde05306200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T22:57:32.650 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:32] ENGINE Bus STARTING 2024-03-11T22:57:32.951 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: CherryPy Checker: 2024-03-11T22:57:32.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: The Application mounted at '' has an empty config. 2024-03-11T22:57:32.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:57:32.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:32] ENGINE Serving on http://:::9283 2024-03-11T22:57:32.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:32 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:32] ENGINE Bus STARTED 2024-03-11T22:57:33.350 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: Standby manager daemon x restarted 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: Standby manager daemon x started 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: Active manager daemon y restarted 2024-03-11T22:57:33.434 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: Activating manager daemon y 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: mgrmap e19: y(active, starting, since 0.013192s), standbys: x, z 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:33.435 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: Manager daemon y is now available 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:33.436 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:33.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:33.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:57:33.437 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:57:33.503 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T22:57:33.503 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: Standby manager daemon x restarted 2024-03-11T22:57:33.503 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: Standby manager daemon x started 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: Active manager daemon y restarted 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: Activating manager daemon y 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: mgrmap e19: y(active, starting, since 0.013192s), standbys: x, z 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:57:33.504 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: Manager daemon y is now available 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.505 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:33.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:33.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:33.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:33.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:57:33.506 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:57:33.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: Standby manager daemon x restarted 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: Standby manager daemon x started 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.? 172.21.15.131:0/558107530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: Active manager daemon y restarted 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: Activating manager daemon y 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: osdmap e31: 3 total, 3 up, 3 in 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: mgrmap e19: y(active, starting, since 0.013192s), standbys: x, z 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T22:57:33.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: Manager daemon y is now available 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.580 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]: dispatch 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi195", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi131", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"}]': finished 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T22:57:33.581 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: [11/Mar/2024:22:57:33] ENGINE Bus STARTING 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: [11/Mar/2024:22:57:33] ENGINE Serving on http://172.21.15.107:8765 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: mgrmap e20: y(active, since 1.01681s), standbys: x, z 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.284 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.285 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:34 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi107", "caps": []}]: dispatch 2024-03-11T22:57:34.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-11T22:57:34.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: [11/Mar/2024:22:57:33] ENGINE Bus STARTING 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: [11/Mar/2024:22:57:33] ENGINE Serving on http://172.21.15.107:8765 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: mgrmap e20: y(active, since 1.01681s), standbys: x, z 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.330 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.330 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:34 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi107", "caps": []}]: dispatch 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: Updating smithi107:/etc/ceph/ceph.conf 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: Updating smithi131:/etc/ceph/ceph.conf 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: Updating smithi195:/etc/ceph/ceph.conf 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: [11/Mar/2024:22:57:33] ENGINE Bus STARTING 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: [11/Mar/2024:22:57:33] ENGINE Serving on http://172.21.15.107:8765 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: mgrmap e20: y(active, since 1.01681s), standbys: x, z 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:34.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:34 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi107", "caps": []}]: dispatch 2024-03-11T22:57:35.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: [11/Mar/2024:22:57:33] ENGINE Serving on https://172.21.15.107:7151 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: [11/Mar/2024:22:57:33] ENGINE Bus STARTED 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:35 smithi107 ceph-mon[25849]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:57:35.484 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.484 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: [11/Mar/2024:22:57:33] ENGINE Serving on https://172.21.15.107:7151 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: [11/Mar/2024:22:57:33] ENGINE Bus STARTED 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.485 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:35 smithi195 ceph-mon[27640]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: Updating smithi131:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: Updating smithi195:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: [11/Mar/2024:22:57:33] ENGINE Serving on https://172.21.15.107:7151 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: [11/Mar/2024:22:57:33] ENGINE Bus STARTED 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: Updating smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: Updating smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: Updating smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T22:57:35.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:35 smithi131 ceph-mon[27544]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:57:36.292 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:36.292 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]: dispatch 2024-03-11T22:57:36.292 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T22:57:36.292 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.292 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.292 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.293 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:57:36.293 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.293 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.293 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:36.293 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:36 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]: dispatch 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:57:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:36.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:36 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:57:36.381 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi131", "caps": []}]: dispatch 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: mgrmap e21: y(active, since 3s), standbys: x, z 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:36.382 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:36 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi195", "caps": []}]: dispatch 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:37.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:37 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.486 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:37.487 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:37.487 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:37 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:37.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:37 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:37.811 INFO:teuthology.orchestra.run.smithi107.stdout:34359738390 2024-03-11T22:57:37.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T22:57:37.885 INFO:teuthology.orchestra.run.smithi107.stdout:77309411340 2024-03-11T22:57:37.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T22:57:38.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:38.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:57:38.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: mgrmap e22: y(active, since 5s), standbys: x, z 2024-03-11T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:38.579 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.400 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.401 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:39 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:39 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:39.823 INFO:teuthology.orchestra.run.smithi107.stdout:55834574866 2024-03-11T22:57:39.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T22:57:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:39.852 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:40.095 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ::ffff:172.21.15.131 - - [11/Mar/2024:22:57:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:40.441 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:40 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.703 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T22:57:40.704 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.704 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.704 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.704 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:40.704 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:40.704 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:40.704 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:40 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T22:57:40.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:40.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:57:40.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:57:40.811 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:57:40.811 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:40 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.359 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:41.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:41 smithi107 ceph-mon[25849]: Deploying daemon alertmanager.a on smithi107 2024-03-11T22:57:41.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:41 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:41 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.369 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:41 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:41 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:41 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.370 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:41 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.466 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:41 smithi131 ceph-mon[27544]: Deploying daemon alertmanager.a on smithi107 2024-03-11T22:57:41.466 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:41 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.466 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:41 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.466 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:41 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.467 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:41 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.467 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:41 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.467 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:41 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:41 smithi195 ceph-mon[27640]: Deploying daemon alertmanager.a on smithi107 2024-03-11T22:57:41.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:41 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:41 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:41 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:41 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:41 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:41.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:41 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.600 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:42 smithi107 ceph-mon[25849]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:57:42.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:42 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:42 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.601 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:42 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:42 smithi131 ceph-mon[27544]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:57:42.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:42 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:42 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:42 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:42 smithi195 ceph-mon[27640]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-11T22:57:42.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:42 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:42 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:42.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:42 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:44.045 INFO:teuthology.orchestra.run.smithi107.stdout:34359738391 2024-03-11T22:57:44.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:44 smithi107 ceph-mon[25849]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:57:44.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:44 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3881194084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:57:44.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:44 smithi131 ceph-mon[27544]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:57:44.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:44 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3881194084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:57:44.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:44 smithi195 ceph-mon[27640]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T22:57:44.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:44 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3881194084' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T22:57:44.970 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738390 got 34359738391 for osd.0 2024-03-11T22:57:44.970 DEBUG:teuthology.parallel:result is None 2024-03-11T22:57:46.452 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:46 smithi107 ceph-mon[25849]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:57:46.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:46 smithi131 ceph-mon[27544]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:57:46.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:46 smithi195 ceph-mon[27640]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-11T22:57:46.865 INFO:teuthology.orchestra.run.smithi107.stdout:55834574867 2024-03-11T22:57:47.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:47 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3447021063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:57:47.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:47 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3447021063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:57:47.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:47 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3447021063' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T22:57:48.015 INFO:teuthology.orchestra.run.smithi107.stdout:77309411342 2024-03-11T22:57:48.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:48 smithi107 ceph-mon[25849]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:57:48.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:48 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:48.667 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:48 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/3199356254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:57:48.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:48 smithi131 ceph-mon[27544]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:57:48.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:48 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:48.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:48 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/3199356254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:57:48.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:48 smithi195 ceph-mon[27640]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:57:48.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:48 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:57:48.829 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:48 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/3199356254' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T22:57:48.984 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574866 got 55834574867 for osd.1 2024-03-11T22:57:48.984 DEBUG:teuthology.parallel:result is None 2024-03-11T22:57:49.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:49 smithi107 ceph-mon[25849]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:57:49.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:49 smithi131 ceph-mon[27544]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:57:49.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:49 smithi195 ceph-mon[27640]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-03-11T22:57:50.063 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:49 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ::ffff:172.21.15.131 - - [11/Mar/2024:22:57:49] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:57:50.064 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411340 got 77309411342 for osd.2 2024-03-11T22:57:50.064 DEBUG:teuthology.parallel:result is None 2024-03-11T22:57:50.064 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T22:57:50.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:57:51.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:51 smithi131 ceph-mon[27544]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:51.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:51 smithi107 ceph-mon[25849]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:51.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:51 smithi195 ceph-mon[27640]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:52.058 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:53.550 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:53 smithi107 podman[48852]: 2024-03-11 22:57:53.424148949 +0000 UTC m=+0.202685490 container create 9a55d7369b8e2dcb38b21eb36c9932b695f374ff38a0b46d42ca96f4ce3a22c3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:57:53.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:53 smithi131 ceph-mon[27544]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:53.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:53 smithi195 ceph-mon[27640]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:53.862 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:53 smithi107 ceph-mon[25849]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:53 smithi107 podman[48852]: 2024-03-11 22:57:53.970971779 +0000 UTC m=+0.749508292 container init 9a55d7369b8e2dcb38b21eb36c9932b695f374ff38a0b46d42ca96f4ce3a22c3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:53 smithi107 podman[48852]: 2024-03-11 22:57:53.976887878 +0000 UTC m=+0.755424373 container start 9a55d7369b8e2dcb38b21eb36c9932b695f374ff38a0b46d42ca96f4ce3a22c3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:53 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:53.988Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:53 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:53.988Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:53 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:53.990Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.107 port=9094 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:53 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:53.991Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:54.014Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:57:54.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:54.014Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:57:54.203 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:54.017Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T22:57:54.203 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:54.017Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T22:57:54.701 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:54 smithi107 bash[48852]: 9a55d7369b8e2dcb38b21eb36c9932b695f374ff38a0b46d42ca96f4ce3a22c3 2024-03-11T22:57:54.702 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:54 smithi107 systemd[1]: Started Ceph alertmanager.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:57:54.745 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:57:54.747 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-03-11T22:57:55.176 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:54] ENGINE Bus STOPPING 2024-03-11T22:57:55.176 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:57:55.177 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:54] ENGINE Bus STOPPED 2024-03-11T22:57:55.177 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:54] ENGINE Bus STARTING 2024-03-11T22:57:55.177 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:54] ENGINE Serving on http://:::9283 2024-03-11T22:57:55.177 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:54 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:57:54] ENGINE Bus STARTED 2024-03-11T22:57:55.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:55.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:55.810 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:55.811 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-mon[25849]: Deploying daemon grafana.a on smithi131 2024-03-11T22:57:55.838 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-03-11T22:57:54.490741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":100772,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":281122396,"statfs":{"total":287972524032,"available":287869333504,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002884"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476694+0000","last_change":"2024-03-11T22:57:09.762913+0000","last_active":"2024-03-11T22:57:32.476694+0000","last_peered":"2024-03-11T22:57:32.476694+0000","last_clean":"2024-03-11T22:57:32.476694+0000","last_became_active":"2024-03-11T22:57:09.760726+0000","last_became_peered":"2024-03-11T22:57:09.760726+0000","last_unstale":"2024-03-11T22:57:32.476694+0000","last_undegraded":"2024-03-11T22:57:32.476694+0000","last_fullsized":"2024-03-11T22:57:32.476694+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:53.267775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474068+0000","last_change":"2024-03-11T22:57:07.749096+0000","last_active":"2024-03-11T22:57:32.474068+0000","last_peered":"2024-03-11T22:57:32.474068+0000","last_clean":"2024-03-11T22:57:32.474068+0000","last_became_active":"2024-03-11T22:57:07.748607+0000","last_became_peered":"2024-03-11T22:57:07.748607+0000","last_unstale":"2024-03-11T22:57:32.474068+0000","last_undegraded":"2024-03-11T22:57:32.474068+0000","last_fullsized":"2024-03-11T22:57:32.474068+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:17:20.680348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477357+0000","last_change":"2024-03-11T22:57:03.737085+0000","last_active":"2024-03-11T22:57:32.477357+0000","last_peered":"2024-03-11T22:57:32.477357+0000","last_clean":"2024-03-11T22:57:32.477357+0000","last_became_active":"2024-03-11T22:57:03.736541+0000","last_became_peered":"2024-03-11T22:57:03.736541+0000","last_unstale":"2024-03-11T22:57:32.477357+0000","last_undegraded":"2024-03-11T22:57:32.477357+0000","last_fullsized":"2024-03-11T22:57:32.477357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:34:58.849987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476988+0000","last_change":"2024-03-11T22:57:05.745752+0000","last_active":"2024-03-11T22:57:32.476988+0000","last_peered":"2024-03-11T22:57:32.476988+0000","last_clean":"2024-03-11T22:57:32.476988+0000","last_became_active":"2024-03-11T22:57:05.745397+0000","last_became_peered":"2024-03-11T22:57:05.745397+0000","last_unstale":"2024-03-11T22:57:32.476988+0000","last_undegraded":"2024-03-11T22:57:32.476988+0000","last_fullsized":"2024-03-11T22:57:32.476988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:01.793934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476453+0000","last_change":"2024-03-11T22:57:09.762139+0000","last_active":"2024-03-11T22:57:32.476453+0000","last_peered":"2024-03-11T22:57:32.476453+0000","last_clean":"2024-03-11T22:57:32.476453+0000","last_became_active":"2024-03-11T22:57:09.761618+0000","last_became_peered":"2024-03-11T22:57:09.761618+0000","last_unstale":"2024-03-11T22:57:32.476453+0000","last_undegraded":"2024-03-11T22:57:32.476453+0000","last_fullsized":"2024-03-11T22:57:32.476453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:47.618573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476575+0000","last_change":"2024-03-11T22:57:07.753813+0000","last_active":"2024-03-11T22:57:32.476575+0000","last_peered":"2024-03-11T22:57:32.476575+0000","last_clean":"2024-03-11T22:57:32.476575+0000","last_became_active":"2024-03-11T22:57:07.753404+0000","last_became_peered":"2024-03-11T22:57:07.753404+0000","last_unstale":"2024-03-11T22:57:32.476575+0000","last_undegraded":"2024-03-11T22:57:32.476575+0000","last_fullsized":"2024-03-11T22:57:32.476575+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:02.797889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477334+0000","last_change":"2024-03-11T22:57:03.737412+0000","last_active":"2024-03-11T22:57:32.477334+0000","last_peered":"2024-03-11T22:57:32.477334+0000","last_clean":"2024-03-11T22:57:32.477334+0000","last_became_active":"2024-03-11T22:57:03.737170+0000","last_became_peered":"2024-03-11T22:57:03.737170+0000","last_unstale":"2024-03-11T22:57:32.477334+0000","last_undegraded":"2024-03-11T22:57:32.477334+0000","last_fullsized":"2024-03-11T22:57:32.477334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:08.547803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477243+0000","last_change":"2024-03-11T22:57:05.739542+0000","last_active":"2024-03-11T22:57:32.477243+0000","last_peered":"2024-03-11T22:57:32.477243+0000","last_clean":"2024-03-11T22:57:32.477243+0000","last_became_active":"2024-03-11T22:57:05.739236+0000","last_became_peered":"2024-03-11T22:57:05.739236+0000","last_unstale":"2024-03-11T22:57:32.477243+0000","last_undegraded":"2024-03-11T22:57:32.477243+0000","last_fullsized":"2024-03-11T22:57:32.477243+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:12.390863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474167+0000","last_change":"2024-03-11T22:57:09.759783+0000","last_active":"2024-03-11T22:57:32.474167+0000","last_peered":"2024-03-11T22:57:32.474167+0000","last_clean":"2024-03-11T22:57:32.474167+0000","last_became_active":"2024-03-11T22:57:09.759324+0000","last_became_peered":"2024-03-11T22:57:09.759324+0000","last_unstale":"2024-03-11T22:57:32.474167+0000","last_undegraded":"2024-03-11T22:57:32.474167+0000","last_fullsized":"2024-03-11T22:57:32.474167+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:49.437193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477183+0000","last_change":"2024-03-11T22:57:07.752708+0000","last_active":"2024-03-11T22:57:32.477183+0000","last_peered":"2024-03-11T22:57:32.477183+0000","last_clean":"2024-03-11T22:57:32.477183+0000","last_became_active":"2024-03-11T22:57:07.752392+0000","last_became_peered":"2024-03-11T22:57:07.752392+0000","last_unstale":"2024-03-11T22:57:32.477183+0000","last_undegraded":"2024-03-11T22:57:32.477183+0000","last_fullsized":"2024-03-11T22:57:32.477183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:59:30.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476912+0000","last_change":"2024-03-11T22:57:03.739643+0000","last_active":"2024-03-11T22:57:32.476912+0000","last_peered":"2024-03-11T22:57:32.476912+0000","last_clean":"2024-03-11T22:57:32.476912+0000","last_became_active":"2024-03-11T22:57:03.739360+0000","last_became_peered":"2024-03-11T22:57:03.739360+0000","last_unstale":"2024-03-11T22:57:32.476912+0000","last_undegraded":"2024-03-11T22:57:32.476912+0000","last_fullsized":"2024-03-11T22:57:32.476912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:22.274797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478195+0000","last_change":"2024-03-11T22:57:05.742855+0000","last_active":"2024-03-11T22:57:32.478195+0000","last_peered":"2024-03-11T22:57:32.478195+0000","last_clean":"2024-03-11T22:57:32.478195+0000","last_became_active":"2024-03-11T22:57:05.737925+0000","last_became_peered":"2024-03-11T22:57:05.737925+0000","last_unstale":"2024-03-11T22:57:32.478195+0000","last_undegraded":"2024-03-11T22:57:32.478195+0000","last_fullsized":"2024-03-11T22:57:32.478195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:17:04.665276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476940+0000","last_change":"2024-03-11T22:57:09.760325+0000","last_active":"2024-03-11T22:57:32.476940+0000","last_peered":"2024-03-11T22:57:32.476940+0000","last_clean":"2024-03-11T22:57:32.476940+0000","last_became_active":"2024-03-11T22:57:09.759805+0000","last_became_peered":"2024-03-11T22:57:09.759805+0000","last_unstale":"2024-03-11T22:57:32.476940+0000","last_undegraded":"2024-03-11T22:57:32.476940+0000","last_fullsized":"2024-03-11T22:57:32.476940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:08.840307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476465+0000","last_change":"2024-03-11T22:57:07.753580+0000","last_active":"2024-03-11T22:57:32.476465+0000","last_peered":"2024-03-11T22:57:32.476465+0000","last_clean":"2024-03-11T22:57:32.476465+0000","last_became_active":"2024-03-11T22:57:07.752664+0000","last_became_peered":"2024-03-11T22:57:07.752664+0000","last_unstale":"2024-03-11T22:57:32.476465+0000","last_undegraded":"2024-03-11T22:57:32.476465+0000","last_fullsized":"2024-03-11T22:57:32.476465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:21.044792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475634+0000","last_change":"2024-03-11T22:57:03.738421+0000","last_active":"2024-03-11T22:57:32.475634+0000","last_peered":"2024-03-11T22:57:32.475634+0000","last_clean":"2024-03-11T22:57:32.475634+0000","last_became_active":"2024-03-11T22:57:03.737164+0000","last_became_peered":"2024-03-11T22:57:03.737164+0000","last_unstale":"2024-03-11T22:57:32.475634+0000","last_undegraded":"2024-03-11T22:57:32.475634+0000","last_fullsized":"2024-03-11T22:57:32.475634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:53.298316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476812+0000","last_change":"2024-03-11T22:57:05.742629+0000","last_active":"2024-03-11T22:57:32.476812+0000","last_peered":"2024-03-11T22:57:32.476812+0000","last_clean":"2024-03-11T22:57:32.476812+0000","last_became_active":"2024-03-11T22:57:05.742161+0000","last_became_peered":"2024-03-11T22:57:05.742161+0000","last_unstale":"2024-03-11T22:57:32.476812+0000","last_undegraded":"2024-03-11T22:57:32.476812+0000","last_fullsized":"2024-03-11T22:57:32.476812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:39:15.155874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477438+0000","last_change":"2024-03-11T22:57:09.761128+0000","last_active":"2024-03-11T22:57:32.477438+0000","last_peered":"2024-03-11T22:57:32.477438+0000","last_clean":"2024-03-11T22:57:32.477438+0000","last_became_active":"2024-03-11T22:57:09.760742+0000","last_became_peered":"2024-03-11T22:57:09.760742+0000","last_unstale":"2024-03-11T22:57:32.477438+0000","last_undegraded":"2024-03-11T22:57:32.477438+0000","last_fullsized":"2024-03-11T22:57:32.477438+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:35:38.466536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475396+0000","last_change":"2024-03-11T22:57:07.752251+0000","last_active":"2024-03-11T22:57:32.475396+0000","last_peered":"2024-03-11T22:57:32.475396+0000","last_clean":"2024-03-11T22:57:32.475396+0000","last_became_active":"2024-03-11T22:57:07.751887+0000","last_became_peered":"2024-03-11T22:57:07.751887+0000","last_unstale":"2024-03-11T22:57:32.475396+0000","last_undegraded":"2024-03-11T22:57:32.475396+0000","last_fullsized":"2024-03-11T22:57:32.475396+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.561675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476190+0000","last_change":"2024-03-11T22:57:03.737024+0000","last_active":"2024-03-11T22:57:32.476190+0000","last_peered":"2024-03-11T22:57:32.476190+0000","last_clean":"2024-03-11T22:57:32.476190+0000","last_became_active":"2024-03-11T22:57:03.736640+0000","last_became_peered":"2024-03-11T22:57:03.736640+0000","last_unstale":"2024-03-11T22:57:32.476190+0000","last_undegraded":"2024-03-11T22:57:32.476190+0000","last_fullsized":"2024-03-11T22:57:32.476190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:24.522626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474726+0000","last_change":"2024-03-11T22:57:05.738099+0000","last_active":"2024-03-11T22:57:32.474726+0000","last_peered":"2024-03-11T22:57:32.474726+0000","last_clean":"2024-03-11T22:57:32.474726+0000","last_became_active":"2024-03-11T22:57:05.737792+0000","last_became_peered":"2024-03-11T22:57:05.737792+0000","last_unstale":"2024-03-11T22:57:32.474726+0000","last_undegraded":"2024-03-11T22:57:32.474726+0000","last_fullsized":"2024-03-11T22:57:32.474726+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:42.270356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478237+0000","last_change":"2024-03-11T22:57:09.764894+0000","last_active":"2024-03-11T22:57:32.478237+0000","last_peered":"2024-03-11T22:57:32.478237+0000","last_clean":"2024-03-11T22:57:32.478237+0000","last_became_active":"2024-03-11T22:57:09.764537+0000","last_became_peered":"2024-03-11T22:57:09.764537+0000","last_unstale":"2024-03-11T22:57:32.478237+0000","last_undegraded":"2024-03-11T22:57:32.478237+0000","last_fullsized":"2024-03-11T22:57:32.478237+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:38:24.491688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475537+0000","last_change":"2024-03-11T22:57:07.752347+0000","last_active":"2024-03-11T22:57:32.475537+0000","last_peered":"2024-03-11T22:57:32.475537+0000","last_clean":"2024-03-11T22:57:32.475537+0000","last_became_active":"2024-03-11T22:57:07.751978+0000","last_became_peered":"2024-03-11T22:57:07.751978+0000","last_unstale":"2024-03-11T22:57:32.475537+0000","last_undegraded":"2024-03-11T22:57:32.475537+0000","last_fullsized":"2024-03-11T22:57:32.475537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:27:47.680032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476748+0000","last_change":"2024-03-11T22:57:03.740322+0000","last_active":"2024-03-11T22:57:32.476748+0000","last_peered":"2024-03-11T22:57:32.476748+0000","last_clean":"2024-03-11T22:57:32.476748+0000","last_became_active":"2024-03-11T22:57:03.739334+0000","last_became_peered":"2024-03-11T22:57:03.739334+0000","last_unstale":"2024-03-11T22:57:32.476748+0000","last_undegraded":"2024-03-11T22:57:32.476748+0000","last_fullsized":"2024-03-11T22:57:32.476748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:53:08.190951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475924+0000","last_change":"2024-03-11T22:57:05.738500+0000","last_active":"2024-03-11T22:57:32.475924+0000","last_peered":"2024-03-11T22:57:32.475924+0000","last_clean":"2024-03-11T22:57:32.475924+0000","last_became_active":"2024-03-11T22:57:05.737850+0000","last_became_peered":"2024-03-11T22:57:05.737850+0000","last_unstale":"2024-03-11T22:57:32.475924+0000","last_undegraded":"2024-03-11T22:57:32.475924+0000","last_fullsized":"2024-03-11T22:57:32.475924+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:40:00.898768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476546+0000","last_change":"2024-03-11T22:57:09.764501+0000","last_active":"2024-03-11T22:57:32.476546+0000","last_peered":"2024-03-11T22:57:32.476546+0000","last_clean":"2024-03-11T22:57:32.476546+0000","last_became_active":"2024-03-11T22:57:09.764083+0000","last_became_peered":"2024-03-11T22:57:09.764083+0000","last_unstale":"2024-03-11T22:57:32.476546+0000","last_undegraded":"2024-03-11T22:57:32.476546+0000","last_fullsized":"2024-03-11T22:57:32.476546+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:29:55.057612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789917+0000","last_change":"2024-03-11T22:57:07.750746+0000","last_active":"2024-03-11T22:57:43.789917+0000","last_peered":"2024-03-11T22:57:43.789917+0000","last_clean":"2024-03-11T22:57:43.789917+0000","last_became_active":"2024-03-11T22:57:07.750363+0000","last_became_peered":"2024-03-11T22:57:07.750363+0000","last_unstale":"2024-03-11T22:57:43.789917+0000","last_undegraded":"2024-03-11T22:57:43.789917+0000","last_fullsized":"2024-03-11T22:57:43.789917+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:44:21.299494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476609+0000","last_change":"2024-03-11T22:57:03.737407+0000","last_active":"2024-03-11T22:57:32.476609+0000","last_peered":"2024-03-11T22:57:32.476609+0000","last_clean":"2024-03-11T22:57:32.476609+0000","last_became_active":"2024-03-11T22:57:03.736266+0000","last_became_peered":"2024-03-11T22:57:03.736266+0000","last_unstale":"2024-03-11T22:57:32.476609+0000","last_undegraded":"2024-03-11T22:57:32.476609+0000","last_fullsized":"2024-03-11T22:57:32.476609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:08.834158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475493+0000","last_change":"2024-03-11T22:57:05.740586+0000","last_active":"2024-03-11T22:57:32.475493+0000","last_peered":"2024-03-11T22:57:32.475493+0000","last_clean":"2024-03-11T22:57:32.475493+0000","last_became_active":"2024-03-11T22:57:05.739748+0000","last_became_peered":"2024-03-11T22:57:05.739748+0000","last_unstale":"2024-03-11T22:57:32.475493+0000","last_undegraded":"2024-03-11T22:57:32.475493+0000","last_fullsized":"2024-03-11T22:57:32.475493+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:50:28.256620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477960+0000","last_change":"2024-03-11T22:57:09.763259+0000","last_active":"2024-03-11T22:57:32.477960+0000","last_peered":"2024-03-11T22:57:32.477960+0000","last_clean":"2024-03-11T22:57:32.477960+0000","last_became_active":"2024-03-11T22:57:09.762419+0000","last_became_peered":"2024-03-11T22:57:09.762419+0000","last_unstale":"2024-03-11T22:57:32.477960+0000","last_undegraded":"2024-03-11T22:57:32.477960+0000","last_fullsized":"2024-03-11T22:57:32.477960+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:20.596774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476761+0000","last_change":"2024-03-11T22:57:07.753313+0000","last_active":"2024-03-11T22:57:32.476761+0000","last_peered":"2024-03-11T22:57:32.476761+0000","last_clean":"2024-03-11T22:57:32.476761+0000","last_became_active":"2024-03-11T22:57:07.752947+0000","last_became_peered":"2024-03-11T22:57:07.752947+0000","last_unstale":"2024-03-11T22:57:32.476761+0000","last_undegraded":"2024-03-11T22:57:32.476761+0000","last_fullsized":"2024-03-11T22:57:32.476761+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:40.135364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474204+0000","last_change":"2024-03-11T22:57:03.739969+0000","last_active":"2024-03-11T22:57:32.474204+0000","last_peered":"2024-03-11T22:57:32.474204+0000","last_clean":"2024-03-11T22:57:32.474204+0000","last_became_active":"2024-03-11T22:57:03.739626+0000","last_became_peered":"2024-03-11T22:57:03.739626+0000","last_unstale":"2024-03-11T22:57:32.474204+0000","last_undegraded":"2024-03-11T22:57:32.474204+0000","last_fullsized":"2024-03-11T22:57:32.474204+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:56:16.974454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475759+0000","last_change":"2024-03-11T22:57:05.740010+0000","last_active":"2024-03-11T22:57:32.475759+0000","last_peered":"2024-03-11T22:57:32.475759+0000","last_clean":"2024-03-11T22:57:32.475759+0000","last_became_active":"2024-03-11T22:57:05.739620+0000","last_became_peered":"2024-03-11T22:57:05.739620+0000","last_unstale":"2024-03-11T22:57:32.475759+0000","last_undegraded":"2024-03-11T22:57:32.475759+0000","last_fullsized":"2024-03-11T22:57:32.475759+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:04.313850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476800+0000","last_change":"2024-03-11T22:57:09.762698+0000","last_active":"2024-03-11T22:57:32.476800+0000","last_peered":"2024-03-11T22:57:32.476800+0000","last_clean":"2024-03-11T22:57:32.476800+0000","last_became_active":"2024-03-11T22:57:09.761114+0000","last_became_peered":"2024-03-11T22:57:09.761114+0000","last_unstale":"2024-03-11T22:57:32.476800+0000","last_undegraded":"2024-03-11T22:57:32.476800+0000","last_fullsized":"2024-03-11T22:57:32.476800+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:25.302635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475572+0000","last_change":"2024-03-11T22:57:07.748928+0000","last_active":"2024-03-11T22:57:32.475572+0000","last_peered":"2024-03-11T22:57:32.475572+0000","last_clean":"2024-03-11T22:57:32.475572+0000","last_became_active":"2024-03-11T22:57:07.748477+0000","last_became_peered":"2024-03-11T22:57:07.748477+0000","last_unstale":"2024-03-11T22:57:32.475572+0000","last_undegraded":"2024-03-11T22:57:32.475572+0000","last_fullsized":"2024-03-11T22:57:32.475572+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:40.078029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.479717+0000","last_change":"2024-03-11T22:57:03.737719+0000","last_active":"2024-03-11T22:57:32.479717+0000","last_peered":"2024-03-11T22:57:32.479717+0000","last_clean":"2024-03-11T22:57:32.479717+0000","last_became_active":"2024-03-11T22:57:03.737415+0000","last_became_peered":"2024-03-11T22:57:03.737415+0000","last_unstale":"2024-03-11T22:57:32.479717+0000","last_undegraded":"2024-03-11T22:57:32.479717+0000","last_fullsized":"2024-03-11T22:57:32.479717+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:19.787995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478585+0000","last_change":"2024-03-11T22:57:05.742778+0000","last_active":"2024-03-11T22:57:32.478585+0000","last_peered":"2024-03-11T22:57:32.478585+0000","last_clean":"2024-03-11T22:57:32.478585+0000","last_became_active":"2024-03-11T22:57:05.741510+0000","last_became_peered":"2024-03-11T22:57:05.741510+0000","last_unstale":"2024-03-11T22:57:32.478585+0000","last_undegraded":"2024-03-11T22:57:32.478585+0000","last_fullsized":"2024-03-11T22:57:32.478585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:55.739074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476398+0000","last_change":"2024-03-11T22:57:09.761294+0000","last_active":"2024-03-11T22:57:32.476398+0000","last_peered":"2024-03-11T22:57:32.476398+0000","last_clean":"2024-03-11T22:57:32.476398+0000","last_became_active":"2024-03-11T22:57:09.760101+0000","last_became_peered":"2024-03-11T22:57:09.760101+0000","last_unstale":"2024-03-11T22:57:32.476398+0000","last_undegraded":"2024-03-11T22:57:32.476398+0000","last_fullsized":"2024-03-11T22:57:32.476398+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:06.089314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476270+0000","last_change":"2024-03-11T22:57:07.752492+0000","last_active":"2024-03-11T22:57:32.476270+0000","last_peered":"2024-03-11T22:57:32.476270+0000","last_clean":"2024-03-11T22:57:32.476270+0000","last_became_active":"2024-03-11T22:57:07.752037+0000","last_became_peered":"2024-03-11T22:57:07.752037+0000","last_unstale":"2024-03-11T22:57:32.476270+0000","last_undegraded":"2024-03-11T22:57:32.476270+0000","last_fullsized":"2024-03-11T22:57:32.476270+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:07:21.665829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478166+0000","last_change":"2024-03-11T22:57:03.740156+0000","last_active":"2024-03-11T22:57:32.478166+0000","last_peered":"2024-03-11T22:57:32.478166+0000","last_clean":"2024-03-11T22:57:32.478166+0000","last_became_active":"2024-03-11T22:57:03.739689+0000","last_became_peered":"2024-03-11T22:57:03.739689+0000","last_unstale":"2024-03-11T22:57:32.478166+0000","last_undegraded":"2024-03-11T22:57:32.478166+0000","last_fullsized":"2024-03-11T22:57:32.478166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:53:27.422909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478315+0000","last_change":"2024-03-11T22:57:05.739241+0000","last_active":"2024-03-11T22:57:32.478315+0000","last_peered":"2024-03-11T22:57:32.478315+0000","last_clean":"2024-03-11T22:57:32.478315+0000","last_became_active":"2024-03-11T22:57:05.738788+0000","last_became_peered":"2024-03-11T22:57:05.738788+0000","last_unstale":"2024-03-11T22:57:32.478315+0000","last_undegraded":"2024-03-11T22:57:32.478315+0000","last_fullsized":"2024-03-11T22:57:32.478315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:01.542288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474310+0000","last_change":"2024-03-11T22:57:09.759621+0000","last_active":"2024-03-11T22:57:32.474310+0000","last_peered":"2024-03-11T22:57:32.474310+0000","last_clean":"2024-03-11T22:57:32.474310+0000","last_became_active":"2024-03-11T22:57:09.759124+0000","last_became_peered":"2024-03-11T22:57:09.759124+0000","last_unstale":"2024-03-11T22:57:32.474310+0000","last_undegraded":"2024-03-11T22:57:32.474310+0000","last_fullsized":"2024-03-11T22:57:32.474310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:33.568136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477428+0000","last_change":"2024-03-11T22:57:07.752617+0000","last_active":"2024-03-11T22:57:32.477428+0000","last_peered":"2024-03-11T22:57:32.477428+0000","last_clean":"2024-03-11T22:57:32.477428+0000","last_became_active":"2024-03-11T22:57:07.752173+0000","last_became_peered":"2024-03-11T22:57:07.752173+0000","last_unstale":"2024-03-11T22:57:32.477428+0000","last_undegraded":"2024-03-11T22:57:32.477428+0000","last_fullsized":"2024-03-11T22:57:32.477428+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:46.434943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476179+0000","last_change":"2024-03-11T22:57:03.739212+0000","last_active":"2024-03-11T22:57:32.476179+0000","last_peered":"2024-03-11T22:57:32.476179+0000","last_clean":"2024-03-11T22:57:32.476179+0000","last_became_active":"2024-03-11T22:57:03.738905+0000","last_became_peered":"2024-03-11T22:57:03.738905+0000","last_unstale":"2024-03-11T22:57:32.476179+0000","last_undegraded":"2024-03-11T22:57:32.476179+0000","last_fullsized":"2024-03-11T22:57:32.476179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:36:14.782339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478503+0000","last_change":"2024-03-11T22:57:05.743280+0000","last_active":"2024-03-11T22:57:32.478503+0000","last_peered":"2024-03-11T22:57:32.478503+0000","last_clean":"2024-03-11T22:57:32.478503+0000","last_became_active":"2024-03-11T22:57:05.741303+0000","last_became_peered":"2024-03-11T22:57:05.741303+0000","last_unstale":"2024-03-11T22:57:32.478503+0000","last_undegraded":"2024-03-11T22:57:32.478503+0000","last_fullsized":"2024-03-11T22:57:32.478503+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:00:21.560762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474744+0000","last_change":"2024-03-11T22:57:09.759360+0000","last_active":"2024-03-11T22:57:32.474744+0000","last_peered":"2024-03-11T22:57:32.474744+0000","last_clean":"2024-03-11T22:57:32.474744+0000","last_became_active":"2024-03-11T22:57:09.759008+0000","last_became_peered":"2024-03-11T22:57:09.759008+0000","last_unstale":"2024-03-11T22:57:32.474744+0000","last_undegraded":"2024-03-11T22:57:32.474744+0000","last_fullsized":"2024-03-11T22:57:32.474744+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:16.327494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475781+0000","last_change":"2024-03-11T22:57:07.752427+0000","last_active":"2024-03-11T22:57:32.475781+0000","last_peered":"2024-03-11T22:57:32.475781+0000","last_clean":"2024-03-11T22:57:32.475781+0000","last_became_active":"2024-03-11T22:57:07.752128+0000","last_became_peered":"2024-03-11T22:57:07.752128+0000","last_unstale":"2024-03-11T22:57:32.475781+0000","last_undegraded":"2024-03-11T22:57:32.475781+0000","last_fullsized":"2024-03-11T22:57:32.475781+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:21.108777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477938+0000","last_change":"2024-03-11T22:57:03.740289+0000","last_active":"2024-03-11T22:57:32.477938+0000","last_peered":"2024-03-11T22:57:32.477938+0000","last_clean":"2024-03-11T22:57:32.477938+0000","last_became_active":"2024-03-11T22:57:03.739870+0000","last_became_peered":"2024-03-11T22:57:03.739870+0000","last_unstale":"2024-03-11T22:57:32.477938+0000","last_undegraded":"2024-03-11T22:57:32.477938+0000","last_fullsized":"2024-03-11T22:57:32.477938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:51.700761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475784+0000","last_change":"2024-03-11T22:57:05.742379+0000","last_active":"2024-03-11T22:57:32.475784+0000","last_peered":"2024-03-11T22:57:32.475784+0000","last_clean":"2024-03-11T22:57:32.475784+0000","last_became_active":"2024-03-11T22:57:05.742028+0000","last_became_peered":"2024-03-11T22:57:05.742028+0000","last_unstale":"2024-03-11T22:57:32.475784+0000","last_undegraded":"2024-03-11T22:57:32.475784+0000","last_fullsized":"2024-03-11T22:57:32.475784+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:06.516997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478160+0000","last_change":"2024-03-11T22:57:09.762523+0000","last_active":"2024-03-11T22:57:32.478160+0000","last_peered":"2024-03-11T22:57:32.478160+0000","last_clean":"2024-03-11T22:57:32.478160+0000","last_became_active":"2024-03-11T22:57:09.762127+0000","last_became_peered":"2024-03-11T22:57:09.762127+0000","last_unstale":"2024-03-11T22:57:32.478160+0000","last_undegraded":"2024-03-11T22:57:32.478160+0000","last_fullsized":"2024-03-11T22:57:32.478160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:26:21.943681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789898+0000","last_change":"2024-03-11T22:57:07.750778+0000","last_active":"2024-03-11T22:57:43.789898+0000","last_peered":"2024-03-11T22:57:43.789898+0000","last_clean":"2024-03-11T22:57:43.789898+0000","last_became_active":"2024-03-11T22:57:07.750279+0000","last_became_peered":"2024-03-11T22:57:07.750279+0000","last_unstale":"2024-03-11T22:57:43.789898+0000","last_undegraded":"2024-03-11T22:57:43.789898+0000","last_fullsized":"2024-03-11T22:57:43.789898+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:26:52.969318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475768+0000","last_change":"2024-03-11T22:57:03.736628+0000","last_active":"2024-03-11T22:57:32.475768+0000","last_peered":"2024-03-11T22:57:32.475768+0000","last_clean":"2024-03-11T22:57:32.475768+0000","last_became_active":"2024-03-11T22:57:03.736315+0000","last_became_peered":"2024-03-11T22:57:03.736315+0000","last_unstale":"2024-03-11T22:57:32.475768+0000","last_undegraded":"2024-03-11T22:57:32.475768+0000","last_fullsized":"2024-03-11T22:57:32.475768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:20:16.776864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476090+0000","last_change":"2024-03-11T22:57:05.739121+0000","last_active":"2024-03-11T22:57:32.476090+0000","last_peered":"2024-03-11T22:57:32.476090+0000","last_clean":"2024-03-11T22:57:32.476090+0000","last_became_active":"2024-03-11T22:57:05.738814+0000","last_became_peered":"2024-03-11T22:57:05.738814+0000","last_unstale":"2024-03-11T22:57:32.476090+0000","last_undegraded":"2024-03-11T22:57:32.476090+0000","last_fullsized":"2024-03-11T22:57:32.476090+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.271529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477249+0000","last_change":"2024-03-11T22:57:09.761298+0000","last_active":"2024-03-11T22:57:32.477249+0000","last_peered":"2024-03-11T22:57:32.477249+0000","last_clean":"2024-03-11T22:57:32.477249+0000","last_became_active":"2024-03-11T22:57:09.760246+0000","last_became_peered":"2024-03-11T22:57:09.760246+0000","last_unstale":"2024-03-11T22:57:32.477249+0000","last_undegraded":"2024-03-11T22:57:32.477249+0000","last_fullsized":"2024-03-11T22:57:32.477249+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:43.800496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789886+0000","last_change":"2024-03-11T22:57:07.752518+0000","last_active":"2024-03-11T22:57:43.789886+0000","last_peered":"2024-03-11T22:57:43.789886+0000","last_clean":"2024-03-11T22:57:43.789886+0000","last_became_active":"2024-03-11T22:57:07.752045+0000","last_became_peered":"2024-03-11T22:57:07.752045+0000","last_unstale":"2024-03-11T22:57:43.789886+0000","last_undegraded":"2024-03-11T22:57:43.789886+0000","last_fullsized":"2024-03-11T22:57:43.789886+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:53:37.387595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474500+0000","last_change":"2024-03-11T22:57:03.739275+0000","last_active":"2024-03-11T22:57:32.474500+0000","last_peered":"2024-03-11T22:57:32.474500+0000","last_clean":"2024-03-11T22:57:32.474500+0000","last_became_active":"2024-03-11T22:57:03.738920+0000","last_became_peered":"2024-03-11T22:57:03.738920+0000","last_unstale":"2024-03-11T22:57:32.474500+0000","last_undegraded":"2024-03-11T22:57:32.474500+0000","last_fullsized":"2024-03-11T22:57:32.474500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:56:38.532008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478608+0000","last_change":"2024-03-11T22:57:05.740777+0000","last_active":"2024-03-11T22:57:32.478608+0000","last_peered":"2024-03-11T22:57:32.478608+0000","last_clean":"2024-03-11T22:57:32.478608+0000","last_became_active":"2024-03-11T22:57:05.737954+0000","last_became_peered":"2024-03-11T22:57:05.737954+0000","last_unstale":"2024-03-11T22:57:32.478608+0000","last_undegraded":"2024-03-11T22:57:32.478608+0000","last_fullsized":"2024-03-11T22:57:32.478608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:45:58.403717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477332+0000","last_change":"2024-03-11T22:57:09.762975+0000","last_active":"2024-03-11T22:57:32.477332+0000","last_peered":"2024-03-11T22:57:32.477332+0000","last_clean":"2024-03-11T22:57:32.477332+0000","last_became_active":"2024-03-11T22:57:09.762282+0000","last_became_peered":"2024-03-11T22:57:09.762282+0000","last_unstale":"2024-03-11T22:57:32.477332+0000","last_undegraded":"2024-03-11T22:57:32.477332+0000","last_fullsized":"2024-03-11T22:57:32.477332+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:24.527788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477147+0000","last_change":"2024-03-11T22:57:07.749624+0000","last_active":"2024-03-11T22:57:32.477147+0000","last_peered":"2024-03-11T22:57:32.477147+0000","last_clean":"2024-03-11T22:57:32.477147+0000","last_became_active":"2024-03-11T22:57:07.749256+0000","last_became_peered":"2024-03-11T22:57:07.749256+0000","last_unstale":"2024-03-11T22:57:32.477147+0000","last_undegraded":"2024-03-11T22:57:32.477147+0000","last_fullsized":"2024-03-11T22:57:32.477147+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:53.679709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476614+0000","last_change":"2024-03-11T22:57:03.739904+0000","last_active":"2024-03-11T22:57:32.476614+0000","last_peered":"2024-03-11T22:57:32.476614+0000","last_clean":"2024-03-11T22:57:32.476614+0000","last_became_active":"2024-03-11T22:57:03.739444+0000","last_became_peered":"2024-03-11T22:57:03.739444+0000","last_unstale":"2024-03-11T22:57:32.476614+0000","last_undegraded":"2024-03-11T22:57:32.476614+0000","last_fullsized":"2024-03-11T22:57:32.476614+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:37.314228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476077+0000","last_change":"2024-03-11T22:57:05.739830+0000","last_active":"2024-03-11T22:57:32.476077+0000","last_peered":"2024-03-11T22:57:32.476077+0000","last_clean":"2024-03-11T22:57:32.476077+0000","last_became_active":"2024-03-11T22:57:05.739388+0000","last_became_peered":"2024-03-11T22:57:05.739388+0000","last_unstale":"2024-03-11T22:57:32.476077+0000","last_undegraded":"2024-03-11T22:57:32.476077+0000","last_fullsized":"2024-03-11T22:57:32.476077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:02.238236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476848+0000","last_change":"2024-03-11T22:57:09.765113+0000","last_active":"2024-03-11T22:57:32.476848+0000","last_peered":"2024-03-11T22:57:32.476848+0000","last_clean":"2024-03-11T22:57:32.476848+0000","last_became_active":"2024-03-11T22:57:09.764533+0000","last_became_peered":"2024-03-11T22:57:09.764533+0000","last_unstale":"2024-03-11T22:57:32.476848+0000","last_undegraded":"2024-03-11T22:57:32.476848+0000","last_fullsized":"2024-03-11T22:57:32.476848+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:20.844411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476829+0000","last_change":"2024-03-11T22:57:07.753488+0000","last_active":"2024-03-11T22:57:32.476829+0000","last_peered":"2024-03-11T22:57:32.476829+0000","last_clean":"2024-03-11T22:57:32.476829+0000","last_became_active":"2024-03-11T22:57:07.752952+0000","last_became_peered":"2024-03-11T22:57:07.752952+0000","last_unstale":"2024-03-11T22:57:32.476829+0000","last_undegraded":"2024-03-11T22:57:32.476829+0000","last_fullsized":"2024-03-11T22:57:32.476829+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:04.219143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475038+0000","last_change":"2024-03-11T22:57:03.736790+0000","last_active":"2024-03-11T22:57:32.475038+0000","last_peered":"2024-03-11T22:57:32.475038+0000","last_clean":"2024-03-11T22:57:32.475038+0000","last_became_active":"2024-03-11T22:57:03.735880+0000","last_became_peered":"2024-03-11T22:57:03.735880+0000","last_unstale":"2024-03-11T22:57:32.475038+0000","last_undegraded":"2024-03-11T22:57:32.475038+0000","last_fullsized":"2024-03-11T22:57:32.475038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:47.646016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474487+0000","last_change":"2024-03-11T22:57:05.739112+0000","last_active":"2024-03-11T22:57:32.474487+0000","last_peered":"2024-03-11T22:57:32.474487+0000","last_clean":"2024-03-11T22:57:32.474487+0000","last_became_active":"2024-03-11T22:57:05.738846+0000","last_became_peered":"2024-03-11T22:57:05.738846+0000","last_unstale":"2024-03-11T22:57:32.474487+0000","last_undegraded":"2024-03-11T22:57:32.474487+0000","last_fullsized":"2024-03-11T22:57:32.474487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:05:14.460883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475266+0000","last_change":"2024-03-11T22:57:09.760083+0000","last_active":"2024-03-11T22:57:32.475266+0000","last_peered":"2024-03-11T22:57:32.475266+0000","last_clean":"2024-03-11T22:57:32.475266+0000","last_became_active":"2024-03-11T22:57:09.759794+0000","last_became_peered":"2024-03-11T22:57:09.759794+0000","last_unstale":"2024-03-11T22:57:32.475266+0000","last_undegraded":"2024-03-11T22:57:32.475266+0000","last_fullsized":"2024-03-11T22:57:32.475266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:54:32.316324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476928+0000","last_change":"2024-03-11T22:57:03.736723+0000","last_active":"2024-03-11T22:57:32.476928+0000","last_peered":"2024-03-11T22:57:32.476928+0000","last_clean":"2024-03-11T22:57:32.476928+0000","last_became_active":"2024-03-11T22:57:03.736417+0000","last_became_peered":"2024-03-11T22:57:03.736417+0000","last_unstale":"2024-03-11T22:57:32.476928+0000","last_undegraded":"2024-03-11T22:57:32.476928+0000","last_fullsized":"2024-03-11T22:57:32.476928+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:51.144744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476872+0000","last_change":"2024-03-11T22:57:05.743444+0000","last_active":"2024-03-11T22:57:32.476872+0000","last_peered":"2024-03-11T22:57:32.476872+0000","last_clean":"2024-03-11T22:57:32.476872+0000","last_became_active":"2024-03-11T22:57:05.743208+0000","last_became_peered":"2024-03-11T22:57:05.743208+0000","last_unstale":"2024-03-11T22:57:32.476872+0000","last_undegraded":"2024-03-11T22:57:32.476872+0000","last_fullsized":"2024-03-11T22:57:32.476872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:37:27.199004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789860+0000","last_change":"2024-03-11T22:57:07.749811+0000","last_active":"2024-03-11T22:57:43.789860+0000","last_peered":"2024-03-11T22:57:43.789860+0000","last_clean":"2024-03-11T22:57:43.789860+0000","last_became_active":"2024-03-11T22:57:07.749356+0000","last_became_peered":"2024-03-11T22:57:07.749356+0000","last_unstale":"2024-03-11T22:57:43.789860+0000","last_undegraded":"2024-03-11T22:57:43.789860+0000","last_fullsized":"2024-03-11T22:57:43.789860+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:01:19.014709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476313+0000","last_change":"2024-03-11T22:57:09.763668+0000","last_active":"2024-03-11T22:57:32.476313+0000","last_peered":"2024-03-11T22:57:32.476313+0000","last_clean":"2024-03-11T22:57:32.476313+0000","last_became_active":"2024-03-11T22:57:09.763290+0000","last_became_peered":"2024-03-11T22:57:09.763290+0000","last_unstale":"2024-03-11T22:57:32.476313+0000","last_undegraded":"2024-03-11T22:57:32.476313+0000","last_fullsized":"2024-03-11T22:57:32.476313+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:08:00.259652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478584+0000","last_change":"2024-03-11T22:57:03.735290+0000","last_active":"2024-03-11T22:57:32.478584+0000","last_peered":"2024-03-11T22:57:32.478584+0000","last_clean":"2024-03-11T22:57:32.478584+0000","last_became_active":"2024-03-11T22:57:03.734975+0000","last_became_peered":"2024-03-11T22:57:03.734975+0000","last_unstale":"2024-03-11T22:57:32.478584+0000","last_undegraded":"2024-03-11T22:57:32.478584+0000","last_fullsized":"2024-03-11T22:57:32.478584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:23.621572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789673+0000","last_change":"2024-03-11T22:57:05.739866+0000","last_active":"2024-03-11T22:57:43.789673+0000","last_peered":"2024-03-11T22:57:43.789673+0000","last_clean":"2024-03-11T22:57:43.789673+0000","last_became_active":"2024-03-11T22:57:05.739378+0000","last_became_peered":"2024-03-11T22:57:05.739378+0000","last_unstale":"2024-03-11T22:57:43.789673+0000","last_undegraded":"2024-03-11T22:57:43.789673+0000","last_fullsized":"2024-03-11T22:57:43.789673+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:40:33.108757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474370+0000","last_change":"2024-03-11T22:57:07.751390+0000","last_active":"2024-03-11T22:57:32.474370+0000","last_peered":"2024-03-11T22:57:32.474370+0000","last_clean":"2024-03-11T22:57:32.474370+0000","last_became_active":"2024-03-11T22:57:07.751124+0000","last_became_peered":"2024-03-11T22:57:07.751124+0000","last_unstale":"2024-03-11T22:57:32.474370+0000","last_undegraded":"2024-03-11T22:57:32.474370+0000","last_fullsized":"2024-03-11T22:57:32.474370+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:10.506049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476336+0000","last_change":"2024-03-11T22:57:09.763967+0000","last_active":"2024-03-11T22:57:32.476336+0000","last_peered":"2024-03-11T22:57:32.476336+0000","last_clean":"2024-03-11T22:57:32.476336+0000","last_became_active":"2024-03-11T22:57:09.763447+0000","last_became_peered":"2024-03-11T22:57:09.763447+0000","last_unstale":"2024-03-11T22:57:32.476336+0000","last_undegraded":"2024-03-11T22:57:32.476336+0000","last_fullsized":"2024-03-11T22:57:32.476336+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:51:54.973911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476839+0000","last_change":"2024-03-11T22:57:03.735738+0000","last_active":"2024-03-11T22:57:32.476839+0000","last_peered":"2024-03-11T22:57:32.476839+0000","last_clean":"2024-03-11T22:57:32.476839+0000","last_became_active":"2024-03-11T22:57:03.735048+0000","last_became_peered":"2024-03-11T22:57:03.735048+0000","last_unstale":"2024-03-11T22:57:32.476839+0000","last_undegraded":"2024-03-11T22:57:32.476839+0000","last_fullsized":"2024-03-11T22:57:32.476839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:29.210506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478206+0000","last_change":"2024-03-11T22:57:05.741126+0000","last_active":"2024-03-11T22:57:32.478206+0000","last_peered":"2024-03-11T22:57:32.478206+0000","last_clean":"2024-03-11T22:57:32.478206+0000","last_became_active":"2024-03-11T22:57:05.739807+0000","last_became_peered":"2024-03-11T22:57:05.739807+0000","last_unstale":"2024-03-11T22:57:32.478206+0000","last_undegraded":"2024-03-11T22:57:32.478206+0000","last_fullsized":"2024-03-11T22:57:32.478206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:41:21.142193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478199+0000","last_change":"2024-03-11T22:57:07.752439+0000","last_active":"2024-03-11T22:57:32.478199+0000","last_peered":"2024-03-11T22:57:32.478199+0000","last_clean":"2024-03-11T22:57:32.478199+0000","last_became_active":"2024-03-11T22:57:07.752092+0000","last_became_peered":"2024-03-11T22:57:07.752092+0000","last_unstale":"2024-03-11T22:57:32.478199+0000","last_undegraded":"2024-03-11T22:57:32.478199+0000","last_fullsized":"2024-03-11T22:57:32.478199+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:39.011327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477425+0000","last_change":"2024-03-11T22:57:09.764809+0000","last_active":"2024-03-11T22:57:32.477425+0000","last_peered":"2024-03-11T22:57:32.477425+0000","last_clean":"2024-03-11T22:57:32.477425+0000","last_became_active":"2024-03-11T22:57:09.764193+0000","last_became_peered":"2024-03-11T22:57:09.764193+0000","last_unstale":"2024-03-11T22:57:32.477425+0000","last_undegraded":"2024-03-11T22:57:32.477425+0000","last_fullsized":"2024-03-11T22:57:32.477425+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:50:53.719080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474002+0000","last_change":"2024-03-11T22:57:05.742991+0000","last_active":"2024-03-11T22:57:32.474002+0000","last_peered":"2024-03-11T22:57:32.474002+0000","last_clean":"2024-03-11T22:57:32.474002+0000","last_became_active":"2024-03-11T22:57:05.742609+0000","last_became_peered":"2024-03-11T22:57:05.742609+0000","last_unstale":"2024-03-11T22:57:32.474002+0000","last_undegraded":"2024-03-11T22:57:32.474002+0000","last_fullsized":"2024-03-11T22:57:32.474002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:48:40.260948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475976+0000","last_change":"2024-03-11T22:57:03.736474+0000","last_active":"2024-03-11T22:57:32.475976+0000","last_peered":"2024-03-11T22:57:32.475976+0000","last_clean":"2024-03-11T22:57:32.475976+0000","last_became_active":"2024-03-11T22:57:03.735410+0000","last_became_peered":"2024-03-11T22:57:03.735410+0000","last_unstale":"2024-03-11T22:57:32.475976+0000","last_undegraded":"2024-03-11T22:57:32.475976+0000","last_fullsized":"2024-03-11T22:57:32.475976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:49.633968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477515+0000","last_change":"2024-03-11T22:57:07.748819+0000","last_active":"2024-03-11T22:57:32.477515+0000","last_peered":"2024-03-11T22:57:32.477515+0000","last_clean":"2024-03-11T22:57:32.477515+0000","last_became_active":"2024-03-11T22:57:07.748526+0000","last_became_peered":"2024-03-11T22:57:07.748526+0000","last_unstale":"2024-03-11T22:57:32.477515+0000","last_undegraded":"2024-03-11T22:57:32.477515+0000","last_fullsized":"2024-03-11T22:57:32.477515+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:27.752699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475016+0000","last_change":"2024-03-11T22:57:09.759517+0000","last_active":"2024-03-11T22:57:32.475016+0000","last_peered":"2024-03-11T22:57:32.475016+0000","last_clean":"2024-03-11T22:57:32.475016+0000","last_became_active":"2024-03-11T22:57:09.759184+0000","last_became_peered":"2024-03-11T22:57:09.759184+0000","last_unstale":"2024-03-11T22:57:32.475016+0000","last_undegraded":"2024-03-11T22:57:32.475016+0000","last_fullsized":"2024-03-11T22:57:32.475016+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:47.749653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475874+0000","last_change":"2024-03-11T22:57:05.739628+0000","last_active":"2024-03-11T22:57:32.475874+0000","last_peered":"2024-03-11T22:57:32.475874+0000","last_clean":"2024-03-11T22:57:32.475874+0000","last_became_active":"2024-03-11T22:57:05.739302+0000","last_became_peered":"2024-03-11T22:57:05.739302+0000","last_unstale":"2024-03-11T22:57:32.475874+0000","last_undegraded":"2024-03-11T22:57:32.475874+0000","last_fullsized":"2024-03-11T22:57:32.475874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:24:19.391447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477659+0000","last_change":"2024-03-11T22:57:03.740036+0000","last_active":"2024-03-11T22:57:32.477659+0000","last_peered":"2024-03-11T22:57:32.477659+0000","last_clean":"2024-03-11T22:57:32.477659+0000","last_became_active":"2024-03-11T22:57:03.739677+0000","last_became_peered":"2024-03-11T22:57:03.739677+0000","last_unstale":"2024-03-11T22:57:32.477659+0000","last_undegraded":"2024-03-11T22:57:32.477659+0000","last_fullsized":"2024-03-11T22:57:32.477659+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:19.119317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:34.579443+0000","last_change":"2024-03-11T22:56:50.610051+0000","last_active":"2024-03-11T22:57:34.579443+0000","last_peered":"2024-03-11T22:57:34.579443+0000","last_clean":"2024-03-11T22:57:34.579443+0000","last_became_active":"2024-03-11T22:56:50.609670+0000","last_became_peered":"2024-03-11T22:56:50.609670+0000","last_unstale":"2024-03-11T22:57:34.579443+0000","last_undegraded":"2024-03-11T22:57:34.579443+0000","last_fullsized":"2024-03-11T22:57:34.579443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:56:49.595444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:56:49.595444+0000","last_clean_scrub_stamp":"2024-03-11T22:56:49.595444+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:42.250456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478674+0000","last_change":"2024-03-11T22:57:07.755284+0000","last_active":"2024-03-11T22:57:32.478674+0000","last_peered":"2024-03-11T22:57:32.478674+0000","last_clean":"2024-03-11T22:57:32.478674+0000","last_became_active":"2024-03-11T22:57:07.754245+0000","last_became_peered":"2024-03-11T22:57:07.754245+0000","last_unstale":"2024-03-11T22:57:32.478674+0000","last_undegraded":"2024-03-11T22:57:32.478674+0000","last_fullsized":"2024-03-11T22:57:32.478674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:17:29.004280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478663+0000","last_change":"2024-03-11T22:57:09.762845+0000","last_active":"2024-03-11T22:57:32.478663+0000","last_peered":"2024-03-11T22:57:32.478663+0000","last_clean":"2024-03-11T22:57:32.478663+0000","last_became_active":"2024-03-11T22:57:09.761497+0000","last_became_peered":"2024-03-11T22:57:09.761497+0000","last_unstale":"2024-03-11T22:57:32.478663+0000","last_undegraded":"2024-03-11T22:57:32.478663+0000","last_fullsized":"2024-03-11T22:57:32.478663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:30:27.509349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476690+0000","last_change":"2024-03-11T22:57:05.740094+0000","last_active":"2024-03-11T22:57:32.476690+0000","last_peered":"2024-03-11T22:57:32.476690+0000","last_clean":"2024-03-11T22:57:32.476690+0000","last_became_active":"2024-03-11T22:57:05.739778+0000","last_became_peered":"2024-03-11T22:57:05.739778+0000","last_unstale":"2024-03-11T22:57:32.476690+0000","last_undegraded":"2024-03-11T22:57:32.476690+0000","last_fullsized":"2024-03-11T22:57:32.476690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:54:45.526939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476100+0000","last_change":"2024-03-11T22:57:03.740222+0000","last_active":"2024-03-11T22:57:32.476100+0000","last_peered":"2024-03-11T22:57:32.476100+0000","last_clean":"2024-03-11T22:57:32.476100+0000","last_became_active":"2024-03-11T22:57:03.739551+0000","last_became_peered":"2024-03-11T22:57:03.739551+0000","last_unstale":"2024-03-11T22:57:32.476100+0000","last_undegraded":"2024-03-11T22:57:32.476100+0000","last_fullsized":"2024-03-11T22:57:32.476100+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:41:53.539081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475513+0000","last_change":"2024-03-11T22:57:07.751750+0000","last_active":"2024-03-11T22:57:32.475513+0000","last_peered":"2024-03-11T22:57:32.475513+0000","last_clean":"2024-03-11T22:57:32.475513+0000","last_became_active":"2024-03-11T22:57:07.751405+0000","last_became_peered":"2024-03-11T22:57:07.751405+0000","last_unstale":"2024-03-11T22:57:32.475513+0000","last_undegraded":"2024-03-11T22:57:32.475513+0000","last_fullsized":"2024-03-11T22:57:32.475513+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:02.426990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476683+0000","last_change":"2024-03-11T22:57:09.762402+0000","last_active":"2024-03-11T22:57:32.476683+0000","last_peered":"2024-03-11T22:57:32.476683+0000","last_clean":"2024-03-11T22:57:32.476683+0000","last_became_active":"2024-03-11T22:57:09.761136+0000","last_became_peered":"2024-03-11T22:57:09.761136+0000","last_unstale":"2024-03-11T22:57:32.476683+0000","last_undegraded":"2024-03-11T22:57:32.476683+0000","last_fullsized":"2024-03-11T22:57:32.476683+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:42:24.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475987+0000","last_change":"2024-03-11T22:57:03.738080+0000","last_active":"2024-03-11T22:57:32.475987+0000","last_peered":"2024-03-11T22:57:32.475987+0000","last_clean":"2024-03-11T22:57:32.475987+0000","last_became_active":"2024-03-11T22:57:03.737766+0000","last_became_peered":"2024-03-11T22:57:03.737766+0000","last_unstale":"2024-03-11T22:57:32.475987+0000","last_undegraded":"2024-03-11T22:57:32.475987+0000","last_fullsized":"2024-03-11T22:57:32.475987+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:38:40.674857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":81,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789477+0000","last_change":"2024-03-11T22:57:05.745716+0000","last_active":"2024-03-11T22:57:43.789477+0000","last_peered":"2024-03-11T22:57:43.789477+0000","last_clean":"2024-03-11T22:57:43.789477+0000","last_became_active":"2024-03-11T22:57:05.745385+0000","last_became_peered":"2024-03-11T22:57:05.745385+0000","last_unstale":"2024-03-11T22:57:43.789477+0000","last_undegraded":"2024-03-11T22:57:43.789477+0000","last_fullsized":"2024-03-11T22:57:43.789477+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:03.189221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789909+0000","last_change":"2024-03-11T22:57:07.748406+0000","last_active":"2024-03-11T22:57:43.789909+0000","last_peered":"2024-03-11T22:57:43.789909+0000","last_clean":"2024-03-11T22:57:43.789909+0000","last_became_active":"2024-03-11T22:57:07.748118+0000","last_became_peered":"2024-03-11T22:57:07.748118+0000","last_unstale":"2024-03-11T22:57:43.789909+0000","last_undegraded":"2024-03-11T22:57:43.789909+0000","last_fullsized":"2024-03-11T22:57:43.789909+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:14.025989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476133+0000","last_change":"2024-03-11T22:57:09.760828+0000","last_active":"2024-03-11T22:57:32.476133+0000","last_peered":"2024-03-11T22:57:32.476133+0000","last_clean":"2024-03-11T22:57:32.476133+0000","last_became_active":"2024-03-11T22:57:09.760424+0000","last_became_peered":"2024-03-11T22:57:09.760424+0000","last_unstale":"2024-03-11T22:57:32.476133+0000","last_undegraded":"2024-03-11T22:57:32.476133+0000","last_fullsized":"2024-03-11T22:57:32.476133+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:23:32.505783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476926+0000","last_change":"2024-03-11T22:57:03.736375+0000","last_active":"2024-03-11T22:57:32.476926+0000","last_peered":"2024-03-11T22:57:32.476926+0000","last_clean":"2024-03-11T22:57:32.476926+0000","last_became_active":"2024-03-11T22:57:03.736083+0000","last_became_peered":"2024-03-11T22:57:03.736083+0000","last_unstale":"2024-03-11T22:57:32.476926+0000","last_undegraded":"2024-03-11T22:57:32.476926+0000","last_fullsized":"2024-03-11T22:57:32.476926+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:10:18.815045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478233+0000","last_change":"2024-03-11T22:57:05.738926+0000","last_active":"2024-03-11T22:57:32.478233+0000","last_peered":"2024-03-11T22:57:32.478233+0000","last_clean":"2024-03-11T22:57:32.478233+0000","last_became_active":"2024-03-11T22:57:05.738589+0000","last_became_peered":"2024-03-11T22:57:05.738589+0000","last_unstale":"2024-03-11T22:57:32.478233+0000","last_undegraded":"2024-03-11T22:57:32.478233+0000","last_fullsized":"2024-03-11T22:57:32.478233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:25:32.229745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789524+0000","last_change":"2024-03-11T22:57:07.753158+0000","last_active":"2024-03-11T22:57:43.789524+0000","last_peered":"2024-03-11T22:57:43.789524+0000","last_clean":"2024-03-11T22:57:43.789524+0000","last_became_active":"2024-03-11T22:57:07.752882+0000","last_became_peered":"2024-03-11T22:57:07.752882+0000","last_unstale":"2024-03-11T22:57:43.789524+0000","last_undegraded":"2024-03-11T22:57:43.789524+0000","last_fullsized":"2024-03-11T22:57:43.789524+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:39.603552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475237+0000","last_change":"2024-03-11T22:57:09.760445+0000","last_active":"2024-03-11T22:57:32.475237+0000","last_peered":"2024-03-11T22:57:32.475237+0000","last_clean":"2024-03-11T22:57:32.475237+0000","last_became_active":"2024-03-11T22:57:09.760136+0000","last_became_peered":"2024-03-11T22:57:09.760136+0000","last_unstale":"2024-03-11T22:57:32.475237+0000","last_undegraded":"2024-03-11T22:57:32.475237+0000","last_fullsized":"2024-03-11T22:57:32.475237+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:27.534194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.479039+0000","last_change":"2024-03-11T22:57:03.737827+0000","last_active":"2024-03-11T22:57:32.479039+0000","last_peered":"2024-03-11T22:57:32.479039+0000","last_clean":"2024-03-11T22:57:32.479039+0000","last_became_active":"2024-03-11T22:57:03.737544+0000","last_became_peered":"2024-03-11T22:57:03.737544+0000","last_unstale":"2024-03-11T22:57:32.479039+0000","last_undegraded":"2024-03-11T22:57:32.479039+0000","last_fullsized":"2024-03-11T22:57:32.479039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:18:32.149946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477134+0000","last_change":"2024-03-11T22:57:05.745719+0000","last_active":"2024-03-11T22:57:32.477134+0000","last_peered":"2024-03-11T22:57:32.477134+0000","last_clean":"2024-03-11T22:57:32.477134+0000","last_became_active":"2024-03-11T22:57:05.745306+0000","last_became_peered":"2024-03-11T22:57:05.745306+0000","last_unstale":"2024-03-11T22:57:32.477134+0000","last_undegraded":"2024-03-11T22:57:32.477134+0000","last_fullsized":"2024-03-11T22:57:32.477134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:48:18.544997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474181+0000","last_change":"2024-03-11T22:57:07.748519+0000","last_active":"2024-03-11T22:57:32.474181+0000","last_peered":"2024-03-11T22:57:32.474181+0000","last_clean":"2024-03-11T22:57:32.474181+0000","last_became_active":"2024-03-11T22:57:07.748216+0000","last_became_peered":"2024-03-11T22:57:07.748216+0000","last_unstale":"2024-03-11T22:57:32.474181+0000","last_undegraded":"2024-03-11T22:57:32.474181+0000","last_fullsized":"2024-03-11T22:57:32.474181+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:39.913218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478774+0000","last_change":"2024-03-11T22:57:09.763248+0000","last_active":"2024-03-11T22:57:32.478774+0000","last_peered":"2024-03-11T22:57:32.478774+0000","last_clean":"2024-03-11T22:57:32.478774+0000","last_became_active":"2024-03-11T22:57:09.762651+0000","last_became_peered":"2024-03-11T22:57:09.762651+0000","last_unstale":"2024-03-11T22:57:32.478774+0000","last_undegraded":"2024-03-11T22:57:32.478774+0000","last_fullsized":"2024-03-11T22:57:32.478774+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:25:00.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789721+0000","last_change":"2024-03-11T22:57:07.752478+0000","last_active":"2024-03-11T22:57:43.789721+0000","last_peered":"2024-03-11T22:57:43.789721+0000","last_clean":"2024-03-11T22:57:43.789721+0000","last_became_active":"2024-03-11T22:57:07.751962+0000","last_became_peered":"2024-03-11T22:57:07.751962+0000","last_unstale":"2024-03-11T22:57:43.789721+0000","last_undegraded":"2024-03-11T22:57:43.789721+0000","last_fullsized":"2024-03-11T22:57:43.789721+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:32.348228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476268+0000","last_change":"2024-03-11T22:57:03.735624+0000","last_active":"2024-03-11T22:57:32.476268+0000","last_peered":"2024-03-11T22:57:32.476268+0000","last_clean":"2024-03-11T22:57:32.476268+0000","last_became_active":"2024-03-11T22:57:03.734510+0000","last_became_peered":"2024-03-11T22:57:03.734510+0000","last_unstale":"2024-03-11T22:57:32.476268+0000","last_undegraded":"2024-03-11T22:57:32.476268+0000","last_fullsized":"2024-03-11T22:57:32.476268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:15:30.311418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474812+0000","last_change":"2024-03-11T22:57:05.738370+0000","last_active":"2024-03-11T22:57:32.474812+0000","last_peered":"2024-03-11T22:57:32.474812+0000","last_clean":"2024-03-11T22:57:32.474812+0000","last_became_active":"2024-03-11T22:57:05.738062+0000","last_became_peered":"2024-03-11T22:57:05.738062+0000","last_unstale":"2024-03-11T22:57:32.474812+0000","last_undegraded":"2024-03-11T22:57:32.474812+0000","last_fullsized":"2024-03-11T22:57:32.474812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:56.868200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475929+0000","last_change":"2024-03-11T22:57:09.760014+0000","last_active":"2024-03-11T22:57:32.475929+0000","last_peered":"2024-03-11T22:57:32.475929+0000","last_clean":"2024-03-11T22:57:32.475929+0000","last_became_active":"2024-03-11T22:57:09.759727+0000","last_became_peered":"2024-03-11T22:57:09.759727+0000","last_unstale":"2024-03-11T22:57:32.475929+0000","last_undegraded":"2024-03-11T22:57:32.475929+0000","last_fullsized":"2024-03-11T22:57:32.475929+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:56:05.619543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477190+0000","last_change":"2024-03-11T22:57:07.749161+0000","last_active":"2024-03-11T22:57:32.477190+0000","last_peered":"2024-03-11T22:57:32.477190+0000","last_clean":"2024-03-11T22:57:32.477190+0000","last_became_active":"2024-03-11T22:57:07.748939+0000","last_became_peered":"2024-03-11T22:57:07.748939+0000","last_unstale":"2024-03-11T22:57:32.477190+0000","last_undegraded":"2024-03-11T22:57:32.477190+0000","last_fullsized":"2024-03-11T22:57:32.477190+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:55.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476516+0000","last_change":"2024-03-11T22:57:03.736236+0000","last_active":"2024-03-11T22:57:32.476516+0000","last_peered":"2024-03-11T22:57:32.476516+0000","last_clean":"2024-03-11T22:57:32.476516+0000","last_became_active":"2024-03-11T22:57:03.735432+0000","last_became_peered":"2024-03-11T22:57:03.735432+0000","last_unstale":"2024-03-11T22:57:32.476516+0000","last_undegraded":"2024-03-11T22:57:32.476516+0000","last_fullsized":"2024-03-11T22:57:32.476516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:16.402095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474978+0000","last_change":"2024-03-11T22:57:05.743022+0000","last_active":"2024-03-11T22:57:32.474978+0000","last_peered":"2024-03-11T22:57:32.474978+0000","last_clean":"2024-03-11T22:57:32.474978+0000","last_became_active":"2024-03-11T22:57:05.742613+0000","last_became_peered":"2024-03-11T22:57:05.742613+0000","last_unstale":"2024-03-11T22:57:32.474978+0000","last_undegraded":"2024-03-11T22:57:32.474978+0000","last_fullsized":"2024-03-11T22:57:32.474978+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:56:10.012331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475276+0000","last_change":"2024-03-11T22:57:09.760368+0000","last_active":"2024-03-11T22:57:32.475276+0000","last_peered":"2024-03-11T22:57:32.475276+0000","last_clean":"2024-03-11T22:57:32.475276+0000","last_became_active":"2024-03-11T22:57:09.759191+0000","last_became_peered":"2024-03-11T22:57:09.759191+0000","last_unstale":"2024-03-11T22:57:32.475276+0000","last_undegraded":"2024-03-11T22:57:32.475276+0000","last_fullsized":"2024-03-11T22:57:32.475276+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:58:01.807812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474701+0000","last_change":"2024-03-11T22:57:03.739018+0000","last_active":"2024-03-11T22:57:32.474701+0000","last_peered":"2024-03-11T22:57:32.474701+0000","last_clean":"2024-03-11T22:57:32.474701+0000","last_became_active":"2024-03-11T22:57:03.738314+0000","last_became_peered":"2024-03-11T22:57:03.738314+0000","last_unstale":"2024-03-11T22:57:32.474701+0000","last_undegraded":"2024-03-11T22:57:32.474701+0000","last_fullsized":"2024-03-11T22:57:32.474701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:41:28.323306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477022+0000","last_change":"2024-03-11T22:57:05.736367+0000","last_active":"2024-03-11T22:57:32.477022+0000","last_peered":"2024-03-11T22:57:32.477022+0000","last_clean":"2024-03-11T22:57:32.477022+0000","last_became_active":"2024-03-11T22:57:05.736118+0000","last_became_peered":"2024-03-11T22:57:05.736118+0000","last_unstale":"2024-03-11T22:57:32.477022+0000","last_undegraded":"2024-03-11T22:57:32.477022+0000","last_fullsized":"2024-03-11T22:57:32.477022+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:55.425498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477548+0000","last_change":"2024-03-11T22:57:07.754093+0000","last_active":"2024-03-11T22:57:32.477548+0000","last_peered":"2024-03-11T22:57:32.477548+0000","last_clean":"2024-03-11T22:57:32.477548+0000","last_became_active":"2024-03-11T22:57:07.753396+0000","last_became_peered":"2024-03-11T22:57:07.753396+0000","last_unstale":"2024-03-11T22:57:32.477548+0000","last_undegraded":"2024-03-11T22:57:32.477548+0000","last_fullsized":"2024-03-11T22:57:32.477548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:11.825147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477467+0000","last_change":"2024-03-11T22:57:09.762171+0000","last_active":"2024-03-11T22:57:32.477467+0000","last_peered":"2024-03-11T22:57:32.477467+0000","last_clean":"2024-03-11T22:57:32.477467+0000","last_became_active":"2024-03-11T22:57:09.761772+0000","last_became_peered":"2024-03-11T22:57:09.761772+0000","last_unstale":"2024-03-11T22:57:32.477467+0000","last_undegraded":"2024-03-11T22:57:32.477467+0000","last_fullsized":"2024-03-11T22:57:32.477467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:31.437327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:43.789716+0000","last_change":"2024-03-11T22:57:07.753577+0000","last_active":"2024-03-11T22:57:43.789716+0000","last_peered":"2024-03-11T22:57:43.789716+0000","last_clean":"2024-03-11T22:57:43.789716+0000","last_became_active":"2024-03-11T22:57:07.752453+0000","last_became_peered":"2024-03-11T22:57:07.752453+0000","last_unstale":"2024-03-11T22:57:43.789716+0000","last_undegraded":"2024-03-11T22:57:43.789716+0000","last_fullsized":"2024-03-11T22:57:43.789716+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:28:10.365849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476075+0000","last_change":"2024-03-11T22:57:03.737879+0000","last_active":"2024-03-11T22:57:32.476075+0000","last_peered":"2024-03-11T22:57:32.476075+0000","last_clean":"2024-03-11T22:57:32.476075+0000","last_became_active":"2024-03-11T22:57:03.737600+0000","last_became_peered":"2024-03-11T22:57:03.737600+0000","last_unstale":"2024-03-11T22:57:32.476075+0000","last_undegraded":"2024-03-11T22:57:32.476075+0000","last_fullsized":"2024-03-11T22:57:32.476075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:27:23.936281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476954+0000","last_change":"2024-03-11T22:57:05.741186+0000","last_active":"2024-03-11T22:57:32.476954+0000","last_peered":"2024-03-11T22:57:32.476954+0000","last_clean":"2024-03-11T22:57:32.476954+0000","last_became_active":"2024-03-11T22:57:05.740675+0000","last_became_peered":"2024-03-11T22:57:05.740675+0000","last_unstale":"2024-03-11T22:57:32.476954+0000","last_undegraded":"2024-03-11T22:57:32.476954+0000","last_fullsized":"2024-03-11T22:57:32.476954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:25:50.580476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475645+0000","last_change":"2024-03-11T22:57:09.760483+0000","last_active":"2024-03-11T22:57:32.475645+0000","last_peered":"2024-03-11T22:57:32.475645+0000","last_clean":"2024-03-11T22:57:32.475645+0000","last_became_active":"2024-03-11T22:57:09.760172+0000","last_became_peered":"2024-03-11T22:57:09.760172+0000","last_unstale":"2024-03-11T22:57:32.475645+0000","last_undegraded":"2024-03-11T22:57:32.475645+0000","last_fullsized":"2024-03-11T22:57:32.475645+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:05.343154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475727+0000","last_change":"2024-03-11T22:57:03.736773+0000","last_active":"2024-03-11T22:57:32.475727+0000","last_peered":"2024-03-11T22:57:32.475727+0000","last_clean":"2024-03-11T22:57:32.475727+0000","last_became_active":"2024-03-11T22:57:03.736443+0000","last_became_peered":"2024-03-11T22:57:03.736443+0000","last_unstale":"2024-03-11T22:57:32.475727+0000","last_undegraded":"2024-03-11T22:57:32.475727+0000","last_fullsized":"2024-03-11T22:57:32.475727+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:42.432197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478674+0000","last_change":"2024-03-11T22:57:05.742940+0000","last_active":"2024-03-11T22:57:32.478674+0000","last_peered":"2024-03-11T22:57:32.478674+0000","last_clean":"2024-03-11T22:57:32.478674+0000","last_became_active":"2024-03-11T22:57:05.741635+0000","last_became_peered":"2024-03-11T22:57:05.741635+0000","last_unstale":"2024-03-11T22:57:32.478674+0000","last_undegraded":"2024-03-11T22:57:32.478674+0000","last_fullsized":"2024-03-11T22:57:32.478674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:04:45.518700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476651+0000","last_change":"2024-03-11T22:57:07.752540+0000","last_active":"2024-03-11T22:57:32.476651+0000","last_peered":"2024-03-11T22:57:32.476651+0000","last_clean":"2024-03-11T22:57:32.476651+0000","last_became_active":"2024-03-11T22:57:07.752124+0000","last_became_peered":"2024-03-11T22:57:07.752124+0000","last_unstale":"2024-03-11T22:57:32.476651+0000","last_undegraded":"2024-03-11T22:57:32.476651+0000","last_fullsized":"2024-03-11T22:57:32.476651+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:08.935596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475570+0000","last_change":"2024-03-11T22:57:09.763961+0000","last_active":"2024-03-11T22:57:32.475570+0000","last_peered":"2024-03-11T22:57:32.475570+0000","last_clean":"2024-03-11T22:57:32.475570+0000","last_became_active":"2024-03-11T22:57:09.763611+0000","last_became_peered":"2024-03-11T22:57:09.763611+0000","last_unstale":"2024-03-11T22:57:32.475570+0000","last_undegraded":"2024-03-11T22:57:32.475570+0000","last_fullsized":"2024-03-11T22:57:32.475570+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:51:06.959552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475674+0000","last_change":"2024-03-11T22:57:03.738159+0000","last_active":"2024-03-11T22:57:32.475674+0000","last_peered":"2024-03-11T22:57:32.475674+0000","last_clean":"2024-03-11T22:57:32.475674+0000","last_became_active":"2024-03-11T22:57:03.737871+0000","last_became_peered":"2024-03-11T22:57:03.737871+0000","last_unstale":"2024-03-11T22:57:32.475674+0000","last_undegraded":"2024-03-11T22:57:32.475674+0000","last_fullsized":"2024-03-11T22:57:32.475674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:48:00.204054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478031+0000","last_change":"2024-03-11T22:57:05.741721+0000","last_active":"2024-03-11T22:57:32.478031+0000","last_peered":"2024-03-11T22:57:32.478031+0000","last_clean":"2024-03-11T22:57:32.478031+0000","last_became_active":"2024-03-11T22:57:05.737966+0000","last_became_peered":"2024-03-11T22:57:05.737966+0000","last_unstale":"2024-03-11T22:57:32.478031+0000","last_undegraded":"2024-03-11T22:57:32.478031+0000","last_fullsized":"2024-03-11T22:57:32.478031+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:01.743914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476448+0000","last_change":"2024-03-11T22:57:07.748770+0000","last_active":"2024-03-11T22:57:32.476448+0000","last_peered":"2024-03-11T22:57:32.476448+0000","last_clean":"2024-03-11T22:57:32.476448+0000","last_became_active":"2024-03-11T22:57:07.748469+0000","last_became_peered":"2024-03-11T22:57:07.748469+0000","last_unstale":"2024-03-11T22:57:32.476448+0000","last_undegraded":"2024-03-11T22:57:32.476448+0000","last_fullsized":"2024-03-11T22:57:32.476448+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:04.628377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476223+0000","last_change":"2024-03-11T22:57:09.760430+0000","last_active":"2024-03-11T22:57:32.476223+0000","last_peered":"2024-03-11T22:57:32.476223+0000","last_clean":"2024-03-11T22:57:32.476223+0000","last_became_active":"2024-03-11T22:57:09.759811+0000","last_became_peered":"2024-03-11T22:57:09.759811+0000","last_unstale":"2024-03-11T22:57:32.476223+0000","last_undegraded":"2024-03-11T22:57:32.476223+0000","last_fullsized":"2024-03-11T22:57:32.476223+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:54:10.600615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477039+0000","last_change":"2024-03-11T22:57:03.736375+0000","last_active":"2024-03-11T22:57:32.477039+0000","last_peered":"2024-03-11T22:57:32.477039+0000","last_clean":"2024-03-11T22:57:32.477039+0000","last_became_active":"2024-03-11T22:57:03.736116+0000","last_became_peered":"2024-03-11T22:57:03.736116+0000","last_unstale":"2024-03-11T22:57:32.477039+0000","last_undegraded":"2024-03-11T22:57:32.477039+0000","last_fullsized":"2024-03-11T22:57:32.477039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:54:59.226299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.479114+0000","last_change":"2024-03-11T22:57:05.743665+0000","last_active":"2024-03-11T22:57:32.479114+0000","last_peered":"2024-03-11T22:57:32.479114+0000","last_clean":"2024-03-11T22:57:32.479114+0000","last_became_active":"2024-03-11T22:57:05.743153+0000","last_became_peered":"2024-03-11T22:57:05.743153+0000","last_unstale":"2024-03-11T22:57:32.479114+0000","last_undegraded":"2024-03-11T22:57:32.479114+0000","last_fullsized":"2024-03-11T22:57:32.479114+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:59:00.702827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475139+0000","last_change":"2024-03-11T22:57:07.752448+0000","last_active":"2024-03-11T22:57:32.475139+0000","last_peered":"2024-03-11T22:57:32.475139+0000","last_clean":"2024-03-11T22:57:32.475139+0000","last_became_active":"2024-03-11T22:57:07.752162+0000","last_became_peered":"2024-03-11T22:57:07.752162+0000","last_unstale":"2024-03-11T22:57:32.475139+0000","last_undegraded":"2024-03-11T22:57:32.475139+0000","last_fullsized":"2024-03-11T22:57:32.475139+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:38.397636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411344,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704692,"statfs":{"total":95990841344,"available":95953604608,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738394,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:57:55.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T22:57:56.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:56.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:57:56.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:57:56.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.061 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:57:56.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:57:56.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.062 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:55 smithi131 ceph-mon[27544]: Deploying daemon grafana.a on smithi131 2024-03-11T22:57:56.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:56.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:56.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:55 smithi195 ceph-mon[27640]: Deploying daemon grafana.a on smithi131 2024-03-11T22:57:56.201 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:57:55 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:57:55.992Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000746791s 2024-03-11T22:57:57.897 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:57 smithi107 ceph-mon[25849]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:57.898 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:57 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:57.898 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:57 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:58.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:57 smithi131 ceph-mon[27544]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:58.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:57 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:58.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:57 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:58.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:57 smithi195 ceph-mon[27640]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:58.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:57 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:58.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:57 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:57:58.178 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:57:59.745 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:57:59 smithi107 ceph-mon[25849]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:57:59.873 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:57:59 smithi195 ceph-mon[27640]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:00.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:57:59 smithi131 ceph-mon[27544]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:00.201 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:57:59 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ::ffff:172.21.15.131 - - [11/Mar/2024:22:57:59] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:58:00.964 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:00 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:01.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:00 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:01.212 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:00 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:01.305 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:58:01.307 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2024-03-11T22:58:01.965 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-03-11T22:58:00.492258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":974,"num_read_kb":821,"num_write":697,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":673,"ondisk_log_size":673,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":100772,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":97659,"kb_avail":281122396,"statfs":{"total":287972524032,"available":287869333504,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":100003151},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002937"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476694+0000","last_change":"2024-03-11T22:57:09.762913+0000","last_active":"2024-03-11T22:57:32.476694+0000","last_peered":"2024-03-11T22:57:32.476694+0000","last_clean":"2024-03-11T22:57:32.476694+0000","last_became_active":"2024-03-11T22:57:09.760726+0000","last_became_peered":"2024-03-11T22:57:09.760726+0000","last_unstale":"2024-03-11T22:57:32.476694+0000","last_undegraded":"2024-03-11T22:57:32.476694+0000","last_fullsized":"2024-03-11T22:57:32.476694+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:41:53.267775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474068+0000","last_change":"2024-03-11T22:57:07.749096+0000","last_active":"2024-03-11T22:57:32.474068+0000","last_peered":"2024-03-11T22:57:32.474068+0000","last_clean":"2024-03-11T22:57:32.474068+0000","last_became_active":"2024-03-11T22:57:07.748607+0000","last_became_peered":"2024-03-11T22:57:07.748607+0000","last_unstale":"2024-03-11T22:57:32.474068+0000","last_undegraded":"2024-03-11T22:57:32.474068+0000","last_fullsized":"2024-03-11T22:57:32.474068+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:17:20.680348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477357+0000","last_change":"2024-03-11T22:57:03.737085+0000","last_active":"2024-03-11T22:57:32.477357+0000","last_peered":"2024-03-11T22:57:32.477357+0000","last_clean":"2024-03-11T22:57:32.477357+0000","last_became_active":"2024-03-11T22:57:03.736541+0000","last_became_peered":"2024-03-11T22:57:03.736541+0000","last_unstale":"2024-03-11T22:57:32.477357+0000","last_undegraded":"2024-03-11T22:57:32.477357+0000","last_fullsized":"2024-03-11T22:57:32.477357+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:34:58.849987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476988+0000","last_change":"2024-03-11T22:57:05.745752+0000","last_active":"2024-03-11T22:57:32.476988+0000","last_peered":"2024-03-11T22:57:32.476988+0000","last_clean":"2024-03-11T22:57:32.476988+0000","last_became_active":"2024-03-11T22:57:05.745397+0000","last_became_peered":"2024-03-11T22:57:05.745397+0000","last_unstale":"2024-03-11T22:57:32.476988+0000","last_undegraded":"2024-03-11T22:57:32.476988+0000","last_fullsized":"2024-03-11T22:57:32.476988+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:34:01.793934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476453+0000","last_change":"2024-03-11T22:57:09.762139+0000","last_active":"2024-03-11T22:57:32.476453+0000","last_peered":"2024-03-11T22:57:32.476453+0000","last_clean":"2024-03-11T22:57:32.476453+0000","last_became_active":"2024-03-11T22:57:09.761618+0000","last_became_peered":"2024-03-11T22:57:09.761618+0000","last_unstale":"2024-03-11T22:57:32.476453+0000","last_undegraded":"2024-03-11T22:57:32.476453+0000","last_fullsized":"2024-03-11T22:57:32.476453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:43:47.618573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476575+0000","last_change":"2024-03-11T22:57:07.753813+0000","last_active":"2024-03-11T22:57:32.476575+0000","last_peered":"2024-03-11T22:57:32.476575+0000","last_clean":"2024-03-11T22:57:32.476575+0000","last_became_active":"2024-03-11T22:57:07.753404+0000","last_became_peered":"2024-03-11T22:57:07.753404+0000","last_unstale":"2024-03-11T22:57:32.476575+0000","last_undegraded":"2024-03-11T22:57:32.476575+0000","last_fullsized":"2024-03-11T22:57:32.476575+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:12:02.797889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477334+0000","last_change":"2024-03-11T22:57:03.737412+0000","last_active":"2024-03-11T22:57:32.477334+0000","last_peered":"2024-03-11T22:57:32.477334+0000","last_clean":"2024-03-11T22:57:32.477334+0000","last_became_active":"2024-03-11T22:57:03.737170+0000","last_became_peered":"2024-03-11T22:57:03.737170+0000","last_unstale":"2024-03-11T22:57:32.477334+0000","last_undegraded":"2024-03-11T22:57:32.477334+0000","last_fullsized":"2024-03-11T22:57:32.477334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:39:08.547803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477243+0000","last_change":"2024-03-11T22:57:05.739542+0000","last_active":"2024-03-11T22:57:32.477243+0000","last_peered":"2024-03-11T22:57:32.477243+0000","last_clean":"2024-03-11T22:57:32.477243+0000","last_became_active":"2024-03-11T22:57:05.739236+0000","last_became_peered":"2024-03-11T22:57:05.739236+0000","last_unstale":"2024-03-11T22:57:32.477243+0000","last_undegraded":"2024-03-11T22:57:32.477243+0000","last_fullsized":"2024-03-11T22:57:32.477243+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:13:12.390863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474167+0000","last_change":"2024-03-11T22:57:09.759783+0000","last_active":"2024-03-11T22:57:32.474167+0000","last_peered":"2024-03-11T22:57:32.474167+0000","last_clean":"2024-03-11T22:57:32.474167+0000","last_became_active":"2024-03-11T22:57:09.759324+0000","last_became_peered":"2024-03-11T22:57:09.759324+0000","last_unstale":"2024-03-11T22:57:32.474167+0000","last_undegraded":"2024-03-11T22:57:32.474167+0000","last_fullsized":"2024-03-11T22:57:32.474167+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:20:49.437193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477183+0000","last_change":"2024-03-11T22:57:07.752708+0000","last_active":"2024-03-11T22:57:32.477183+0000","last_peered":"2024-03-11T22:57:32.477183+0000","last_clean":"2024-03-11T22:57:32.477183+0000","last_became_active":"2024-03-11T22:57:07.752392+0000","last_became_peered":"2024-03-11T22:57:07.752392+0000","last_unstale":"2024-03-11T22:57:32.477183+0000","last_undegraded":"2024-03-11T22:57:32.477183+0000","last_fullsized":"2024-03-11T22:57:32.477183+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:59:30.368993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476912+0000","last_change":"2024-03-11T22:57:03.739643+0000","last_active":"2024-03-11T22:57:32.476912+0000","last_peered":"2024-03-11T22:57:32.476912+0000","last_clean":"2024-03-11T22:57:32.476912+0000","last_became_active":"2024-03-11T22:57:03.739360+0000","last_became_peered":"2024-03-11T22:57:03.739360+0000","last_unstale":"2024-03-11T22:57:32.476912+0000","last_undegraded":"2024-03-11T22:57:32.476912+0000","last_fullsized":"2024-03-11T22:57:32.476912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:22.274797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478195+0000","last_change":"2024-03-11T22:57:05.742855+0000","last_active":"2024-03-11T22:57:32.478195+0000","last_peered":"2024-03-11T22:57:32.478195+0000","last_clean":"2024-03-11T22:57:32.478195+0000","last_became_active":"2024-03-11T22:57:05.737925+0000","last_became_peered":"2024-03-11T22:57:05.737925+0000","last_unstale":"2024-03-11T22:57:32.478195+0000","last_undegraded":"2024-03-11T22:57:32.478195+0000","last_fullsized":"2024-03-11T22:57:32.478195+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:17:04.665276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476940+0000","last_change":"2024-03-11T22:57:09.760325+0000","last_active":"2024-03-11T22:57:32.476940+0000","last_peered":"2024-03-11T22:57:32.476940+0000","last_clean":"2024-03-11T22:57:32.476940+0000","last_became_active":"2024-03-11T22:57:09.759805+0000","last_became_peered":"2024-03-11T22:57:09.759805+0000","last_unstale":"2024-03-11T22:57:32.476940+0000","last_undegraded":"2024-03-11T22:57:32.476940+0000","last_fullsized":"2024-03-11T22:57:32.476940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:16:08.840307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476465+0000","last_change":"2024-03-11T22:57:07.753580+0000","last_active":"2024-03-11T22:57:32.476465+0000","last_peered":"2024-03-11T22:57:32.476465+0000","last_clean":"2024-03-11T22:57:32.476465+0000","last_became_active":"2024-03-11T22:57:07.752664+0000","last_became_peered":"2024-03-11T22:57:07.752664+0000","last_unstale":"2024-03-11T22:57:32.476465+0000","last_undegraded":"2024-03-11T22:57:32.476465+0000","last_fullsized":"2024-03-11T22:57:32.476465+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:50:21.044792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475634+0000","last_change":"2024-03-11T22:57:03.738421+0000","last_active":"2024-03-11T22:57:32.475634+0000","last_peered":"2024-03-11T22:57:32.475634+0000","last_clean":"2024-03-11T22:57:32.475634+0000","last_became_active":"2024-03-11T22:57:03.737164+0000","last_became_peered":"2024-03-11T22:57:03.737164+0000","last_unstale":"2024-03-11T22:57:32.475634+0000","last_undegraded":"2024-03-11T22:57:32.475634+0000","last_fullsized":"2024-03-11T22:57:32.475634+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:57:53.298316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476812+0000","last_change":"2024-03-11T22:57:05.742629+0000","last_active":"2024-03-11T22:57:32.476812+0000","last_peered":"2024-03-11T22:57:32.476812+0000","last_clean":"2024-03-11T22:57:32.476812+0000","last_became_active":"2024-03-11T22:57:05.742161+0000","last_became_peered":"2024-03-11T22:57:05.742161+0000","last_unstale":"2024-03-11T22:57:32.476812+0000","last_undegraded":"2024-03-11T22:57:32.476812+0000","last_fullsized":"2024-03-11T22:57:32.476812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:39:15.155874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477438+0000","last_change":"2024-03-11T22:57:09.761128+0000","last_active":"2024-03-11T22:57:32.477438+0000","last_peered":"2024-03-11T22:57:32.477438+0000","last_clean":"2024-03-11T22:57:32.477438+0000","last_became_active":"2024-03-11T22:57:09.760742+0000","last_became_peered":"2024-03-11T22:57:09.760742+0000","last_unstale":"2024-03-11T22:57:32.477438+0000","last_undegraded":"2024-03-11T22:57:32.477438+0000","last_fullsized":"2024-03-11T22:57:32.477438+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:35:38.466536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475396+0000","last_change":"2024-03-11T22:57:07.752251+0000","last_active":"2024-03-11T22:57:32.475396+0000","last_peered":"2024-03-11T22:57:32.475396+0000","last_clean":"2024-03-11T22:57:32.475396+0000","last_became_active":"2024-03-11T22:57:07.751887+0000","last_became_peered":"2024-03-11T22:57:07.751887+0000","last_unstale":"2024-03-11T22:57:32.475396+0000","last_undegraded":"2024-03-11T22:57:32.475396+0000","last_fullsized":"2024-03-11T22:57:32.475396+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:36:11.561675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476190+0000","last_change":"2024-03-11T22:57:03.737024+0000","last_active":"2024-03-11T22:57:32.476190+0000","last_peered":"2024-03-11T22:57:32.476190+0000","last_clean":"2024-03-11T22:57:32.476190+0000","last_became_active":"2024-03-11T22:57:03.736640+0000","last_became_peered":"2024-03-11T22:57:03.736640+0000","last_unstale":"2024-03-11T22:57:32.476190+0000","last_undegraded":"2024-03-11T22:57:32.476190+0000","last_fullsized":"2024-03-11T22:57:32.476190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:22:24.522626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474726+0000","last_change":"2024-03-11T22:57:05.738099+0000","last_active":"2024-03-11T22:57:32.474726+0000","last_peered":"2024-03-11T22:57:32.474726+0000","last_clean":"2024-03-11T22:57:32.474726+0000","last_became_active":"2024-03-11T22:57:05.737792+0000","last_became_peered":"2024-03-11T22:57:05.737792+0000","last_unstale":"2024-03-11T22:57:32.474726+0000","last_undegraded":"2024-03-11T22:57:32.474726+0000","last_fullsized":"2024-03-11T22:57:32.474726+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:25:42.270356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478237+0000","last_change":"2024-03-11T22:57:09.764894+0000","last_active":"2024-03-11T22:57:32.478237+0000","last_peered":"2024-03-11T22:57:32.478237+0000","last_clean":"2024-03-11T22:57:32.478237+0000","last_became_active":"2024-03-11T22:57:09.764537+0000","last_became_peered":"2024-03-11T22:57:09.764537+0000","last_unstale":"2024-03-11T22:57:32.478237+0000","last_undegraded":"2024-03-11T22:57:32.478237+0000","last_fullsized":"2024-03-11T22:57:32.478237+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:38:24.491688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475537+0000","last_change":"2024-03-11T22:57:07.752347+0000","last_active":"2024-03-11T22:57:32.475537+0000","last_peered":"2024-03-11T22:57:32.475537+0000","last_clean":"2024-03-11T22:57:32.475537+0000","last_became_active":"2024-03-11T22:57:07.751978+0000","last_became_peered":"2024-03-11T22:57:07.751978+0000","last_unstale":"2024-03-11T22:57:32.475537+0000","last_undegraded":"2024-03-11T22:57:32.475537+0000","last_fullsized":"2024-03-11T22:57:32.475537+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:27:47.680032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476748+0000","last_change":"2024-03-11T22:57:03.740322+0000","last_active":"2024-03-11T22:57:32.476748+0000","last_peered":"2024-03-11T22:57:32.476748+0000","last_clean":"2024-03-11T22:57:32.476748+0000","last_became_active":"2024-03-11T22:57:03.739334+0000","last_became_peered":"2024-03-11T22:57:03.739334+0000","last_unstale":"2024-03-11T22:57:32.476748+0000","last_undegraded":"2024-03-11T22:57:32.476748+0000","last_fullsized":"2024-03-11T22:57:32.476748+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:53:08.190951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'5","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475924+0000","last_change":"2024-03-11T22:57:05.738500+0000","last_active":"2024-03-11T22:57:32.475924+0000","last_peered":"2024-03-11T22:57:32.475924+0000","last_clean":"2024-03-11T22:57:32.475924+0000","last_became_active":"2024-03-11T22:57:05.737850+0000","last_became_peered":"2024-03-11T22:57:05.737850+0000","last_unstale":"2024-03-11T22:57:32.475924+0000","last_undegraded":"2024-03-11T22:57:32.475924+0000","last_fullsized":"2024-03-11T22:57:32.475924+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:40:00.898768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476546+0000","last_change":"2024-03-11T22:57:09.764501+0000","last_active":"2024-03-11T22:57:32.476546+0000","last_peered":"2024-03-11T22:57:32.476546+0000","last_clean":"2024-03-11T22:57:32.476546+0000","last_became_active":"2024-03-11T22:57:09.764083+0000","last_became_peered":"2024-03-11T22:57:09.764083+0000","last_unstale":"2024-03-11T22:57:32.476546+0000","last_undegraded":"2024-03-11T22:57:32.476546+0000","last_fullsized":"2024-03-11T22:57:32.476546+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:29:55.057612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.791069+0000","last_change":"2024-03-11T22:57:07.750746+0000","last_active":"2024-03-11T22:57:48.791069+0000","last_peered":"2024-03-11T22:57:48.791069+0000","last_clean":"2024-03-11T22:57:48.791069+0000","last_became_active":"2024-03-11T22:57:07.750363+0000","last_became_peered":"2024-03-11T22:57:07.750363+0000","last_unstale":"2024-03-11T22:57:48.791069+0000","last_undegraded":"2024-03-11T22:57:48.791069+0000","last_fullsized":"2024-03-11T22:57:48.791069+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:44:21.299494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476609+0000","last_change":"2024-03-11T22:57:03.737407+0000","last_active":"2024-03-11T22:57:32.476609+0000","last_peered":"2024-03-11T22:57:32.476609+0000","last_clean":"2024-03-11T22:57:32.476609+0000","last_became_active":"2024-03-11T22:57:03.736266+0000","last_became_peered":"2024-03-11T22:57:03.736266+0000","last_unstale":"2024-03-11T22:57:32.476609+0000","last_undegraded":"2024-03-11T22:57:32.476609+0000","last_fullsized":"2024-03-11T22:57:32.476609+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:02:08.834158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475493+0000","last_change":"2024-03-11T22:57:05.740586+0000","last_active":"2024-03-11T22:57:32.475493+0000","last_peered":"2024-03-11T22:57:32.475493+0000","last_clean":"2024-03-11T22:57:32.475493+0000","last_became_active":"2024-03-11T22:57:05.739748+0000","last_became_peered":"2024-03-11T22:57:05.739748+0000","last_unstale":"2024-03-11T22:57:32.475493+0000","last_undegraded":"2024-03-11T22:57:32.475493+0000","last_fullsized":"2024-03-11T22:57:32.475493+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:50:28.256620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477960+0000","last_change":"2024-03-11T22:57:09.763259+0000","last_active":"2024-03-11T22:57:32.477960+0000","last_peered":"2024-03-11T22:57:32.477960+0000","last_clean":"2024-03-11T22:57:32.477960+0000","last_became_active":"2024-03-11T22:57:09.762419+0000","last_became_peered":"2024-03-11T22:57:09.762419+0000","last_unstale":"2024-03-11T22:57:32.477960+0000","last_undegraded":"2024-03-11T22:57:32.477960+0000","last_fullsized":"2024-03-11T22:57:32.477960+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:20.596774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476761+0000","last_change":"2024-03-11T22:57:07.753313+0000","last_active":"2024-03-11T22:57:32.476761+0000","last_peered":"2024-03-11T22:57:32.476761+0000","last_clean":"2024-03-11T22:57:32.476761+0000","last_became_active":"2024-03-11T22:57:07.752947+0000","last_became_peered":"2024-03-11T22:57:07.752947+0000","last_unstale":"2024-03-11T22:57:32.476761+0000","last_undegraded":"2024-03-11T22:57:32.476761+0000","last_fullsized":"2024-03-11T22:57:32.476761+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:23:40.135364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474204+0000","last_change":"2024-03-11T22:57:03.739969+0000","last_active":"2024-03-11T22:57:32.474204+0000","last_peered":"2024-03-11T22:57:32.474204+0000","last_clean":"2024-03-11T22:57:32.474204+0000","last_became_active":"2024-03-11T22:57:03.739626+0000","last_became_peered":"2024-03-11T22:57:03.739626+0000","last_unstale":"2024-03-11T22:57:32.474204+0000","last_undegraded":"2024-03-11T22:57:32.474204+0000","last_fullsized":"2024-03-11T22:57:32.474204+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:56:16.974454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475759+0000","last_change":"2024-03-11T22:57:05.740010+0000","last_active":"2024-03-11T22:57:32.475759+0000","last_peered":"2024-03-11T22:57:32.475759+0000","last_clean":"2024-03-11T22:57:32.475759+0000","last_became_active":"2024-03-11T22:57:05.739620+0000","last_became_peered":"2024-03-11T22:57:05.739620+0000","last_unstale":"2024-03-11T22:57:32.475759+0000","last_undegraded":"2024-03-11T22:57:32.475759+0000","last_fullsized":"2024-03-11T22:57:32.475759+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:05:04.313850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476800+0000","last_change":"2024-03-11T22:57:09.762698+0000","last_active":"2024-03-11T22:57:32.476800+0000","last_peered":"2024-03-11T22:57:32.476800+0000","last_clean":"2024-03-11T22:57:32.476800+0000","last_became_active":"2024-03-11T22:57:09.761114+0000","last_became_peered":"2024-03-11T22:57:09.761114+0000","last_unstale":"2024-03-11T22:57:32.476800+0000","last_undegraded":"2024-03-11T22:57:32.476800+0000","last_fullsized":"2024-03-11T22:57:32.476800+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:50:25.302635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475572+0000","last_change":"2024-03-11T22:57:07.748928+0000","last_active":"2024-03-11T22:57:32.475572+0000","last_peered":"2024-03-11T22:57:32.475572+0000","last_clean":"2024-03-11T22:57:32.475572+0000","last_became_active":"2024-03-11T22:57:07.748477+0000","last_became_peered":"2024-03-11T22:57:07.748477+0000","last_unstale":"2024-03-11T22:57:32.475572+0000","last_undegraded":"2024-03-11T22:57:32.475572+0000","last_fullsized":"2024-03-11T22:57:32.475572+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:07:40.078029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.479717+0000","last_change":"2024-03-11T22:57:03.737719+0000","last_active":"2024-03-11T22:57:32.479717+0000","last_peered":"2024-03-11T22:57:32.479717+0000","last_clean":"2024-03-11T22:57:32.479717+0000","last_became_active":"2024-03-11T22:57:03.737415+0000","last_became_peered":"2024-03-11T22:57:03.737415+0000","last_unstale":"2024-03-11T22:57:32.479717+0000","last_undegraded":"2024-03-11T22:57:32.479717+0000","last_fullsized":"2024-03-11T22:57:32.479717+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:01:19.787995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478585+0000","last_change":"2024-03-11T22:57:05.742778+0000","last_active":"2024-03-11T22:57:32.478585+0000","last_peered":"2024-03-11T22:57:32.478585+0000","last_clean":"2024-03-11T22:57:32.478585+0000","last_became_active":"2024-03-11T22:57:05.741510+0000","last_became_peered":"2024-03-11T22:57:05.741510+0000","last_unstale":"2024-03-11T22:57:32.478585+0000","last_undegraded":"2024-03-11T22:57:32.478585+0000","last_fullsized":"2024-03-11T22:57:32.478585+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:23:55.739074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476398+0000","last_change":"2024-03-11T22:57:09.761294+0000","last_active":"2024-03-11T22:57:32.476398+0000","last_peered":"2024-03-11T22:57:32.476398+0000","last_clean":"2024-03-11T22:57:32.476398+0000","last_became_active":"2024-03-11T22:57:09.760101+0000","last_became_peered":"2024-03-11T22:57:09.760101+0000","last_unstale":"2024-03-11T22:57:32.476398+0000","last_undegraded":"2024-03-11T22:57:32.476398+0000","last_fullsized":"2024-03-11T22:57:32.476398+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:55:06.089314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476270+0000","last_change":"2024-03-11T22:57:07.752492+0000","last_active":"2024-03-11T22:57:32.476270+0000","last_peered":"2024-03-11T22:57:32.476270+0000","last_clean":"2024-03-11T22:57:32.476270+0000","last_became_active":"2024-03-11T22:57:07.752037+0000","last_became_peered":"2024-03-11T22:57:07.752037+0000","last_unstale":"2024-03-11T22:57:32.476270+0000","last_undegraded":"2024-03-11T22:57:32.476270+0000","last_fullsized":"2024-03-11T22:57:32.476270+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:07:21.665829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478166+0000","last_change":"2024-03-11T22:57:03.740156+0000","last_active":"2024-03-11T22:57:32.478166+0000","last_peered":"2024-03-11T22:57:32.478166+0000","last_clean":"2024-03-11T22:57:32.478166+0000","last_became_active":"2024-03-11T22:57:03.739689+0000","last_became_peered":"2024-03-11T22:57:03.739689+0000","last_unstale":"2024-03-11T22:57:32.478166+0000","last_undegraded":"2024-03-11T22:57:32.478166+0000","last_fullsized":"2024-03-11T22:57:32.478166+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:53:27.422909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'6","reported_seq":32,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478315+0000","last_change":"2024-03-11T22:57:05.739241+0000","last_active":"2024-03-11T22:57:32.478315+0000","last_peered":"2024-03-11T22:57:32.478315+0000","last_clean":"2024-03-11T22:57:32.478315+0000","last_became_active":"2024-03-11T22:57:05.738788+0000","last_became_peered":"2024-03-11T22:57:05.738788+0000","last_unstale":"2024-03-11T22:57:32.478315+0000","last_undegraded":"2024-03-11T22:57:32.478315+0000","last_fullsized":"2024-03-11T22:57:32.478315+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T06:15:01.542288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474310+0000","last_change":"2024-03-11T22:57:09.759621+0000","last_active":"2024-03-11T22:57:32.474310+0000","last_peered":"2024-03-11T22:57:32.474310+0000","last_clean":"2024-03-11T22:57:32.474310+0000","last_became_active":"2024-03-11T22:57:09.759124+0000","last_became_peered":"2024-03-11T22:57:09.759124+0000","last_unstale":"2024-03-11T22:57:32.474310+0000","last_undegraded":"2024-03-11T22:57:32.474310+0000","last_fullsized":"2024-03-11T22:57:32.474310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:44:33.568136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477428+0000","last_change":"2024-03-11T22:57:07.752617+0000","last_active":"2024-03-11T22:57:32.477428+0000","last_peered":"2024-03-11T22:57:32.477428+0000","last_clean":"2024-03-11T22:57:32.477428+0000","last_became_active":"2024-03-11T22:57:07.752173+0000","last_became_peered":"2024-03-11T22:57:07.752173+0000","last_unstale":"2024-03-11T22:57:32.477428+0000","last_undegraded":"2024-03-11T22:57:32.477428+0000","last_fullsized":"2024-03-11T22:57:32.477428+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:46.434943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476179+0000","last_change":"2024-03-11T22:57:03.739212+0000","last_active":"2024-03-11T22:57:32.476179+0000","last_peered":"2024-03-11T22:57:32.476179+0000","last_clean":"2024-03-11T22:57:32.476179+0000","last_became_active":"2024-03-11T22:57:03.738905+0000","last_became_peered":"2024-03-11T22:57:03.738905+0000","last_unstale":"2024-03-11T22:57:32.476179+0000","last_undegraded":"2024-03-11T22:57:32.476179+0000","last_fullsized":"2024-03-11T22:57:32.476179+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:36:14.782339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478503+0000","last_change":"2024-03-11T22:57:05.743280+0000","last_active":"2024-03-11T22:57:32.478503+0000","last_peered":"2024-03-11T22:57:32.478503+0000","last_clean":"2024-03-11T22:57:32.478503+0000","last_became_active":"2024-03-11T22:57:05.741303+0000","last_became_peered":"2024-03-11T22:57:05.741303+0000","last_unstale":"2024-03-11T22:57:32.478503+0000","last_undegraded":"2024-03-11T22:57:32.478503+0000","last_fullsized":"2024-03-11T22:57:32.478503+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:00:21.560762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474744+0000","last_change":"2024-03-11T22:57:09.759360+0000","last_active":"2024-03-11T22:57:32.474744+0000","last_peered":"2024-03-11T22:57:32.474744+0000","last_clean":"2024-03-11T22:57:32.474744+0000","last_became_active":"2024-03-11T22:57:09.759008+0000","last_became_peered":"2024-03-11T22:57:09.759008+0000","last_unstale":"2024-03-11T22:57:32.474744+0000","last_undegraded":"2024-03-11T22:57:32.474744+0000","last_fullsized":"2024-03-11T22:57:32.474744+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:59:16.327494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475781+0000","last_change":"2024-03-11T22:57:07.752427+0000","last_active":"2024-03-11T22:57:32.475781+0000","last_peered":"2024-03-11T22:57:32.475781+0000","last_clean":"2024-03-11T22:57:32.475781+0000","last_became_active":"2024-03-11T22:57:07.752128+0000","last_became_peered":"2024-03-11T22:57:07.752128+0000","last_unstale":"2024-03-11T22:57:32.475781+0000","last_undegraded":"2024-03-11T22:57:32.475781+0000","last_fullsized":"2024-03-11T22:57:32.475781+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:21.108777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477938+0000","last_change":"2024-03-11T22:57:03.740289+0000","last_active":"2024-03-11T22:57:32.477938+0000","last_peered":"2024-03-11T22:57:32.477938+0000","last_clean":"2024-03-11T22:57:32.477938+0000","last_became_active":"2024-03-11T22:57:03.739870+0000","last_became_peered":"2024-03-11T22:57:03.739870+0000","last_unstale":"2024-03-11T22:57:32.477938+0000","last_undegraded":"2024-03-11T22:57:32.477938+0000","last_fullsized":"2024-03-11T22:57:32.477938+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:14:51.700761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475784+0000","last_change":"2024-03-11T22:57:05.742379+0000","last_active":"2024-03-11T22:57:32.475784+0000","last_peered":"2024-03-11T22:57:32.475784+0000","last_clean":"2024-03-11T22:57:32.475784+0000","last_became_active":"2024-03-11T22:57:05.742028+0000","last_became_peered":"2024-03-11T22:57:05.742028+0000","last_unstale":"2024-03-11T22:57:32.475784+0000","last_undegraded":"2024-03-11T22:57:32.475784+0000","last_fullsized":"2024-03-11T22:57:32.475784+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:21:06.516997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478160+0000","last_change":"2024-03-11T22:57:09.762523+0000","last_active":"2024-03-11T22:57:32.478160+0000","last_peered":"2024-03-11T22:57:32.478160+0000","last_clean":"2024-03-11T22:57:32.478160+0000","last_became_active":"2024-03-11T22:57:09.762127+0000","last_became_peered":"2024-03-11T22:57:09.762127+0000","last_unstale":"2024-03-11T22:57:32.478160+0000","last_undegraded":"2024-03-11T22:57:32.478160+0000","last_fullsized":"2024-03-11T22:57:32.478160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:26:21.943681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.791018+0000","last_change":"2024-03-11T22:57:07.750778+0000","last_active":"2024-03-11T22:57:48.791018+0000","last_peered":"2024-03-11T22:57:48.791018+0000","last_clean":"2024-03-11T22:57:48.791018+0000","last_became_active":"2024-03-11T22:57:07.750279+0000","last_became_peered":"2024-03-11T22:57:07.750279+0000","last_unstale":"2024-03-11T22:57:48.791018+0000","last_undegraded":"2024-03-11T22:57:48.791018+0000","last_fullsized":"2024-03-11T22:57:48.791018+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:26:52.969318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475768+0000","last_change":"2024-03-11T22:57:03.736628+0000","last_active":"2024-03-11T22:57:32.475768+0000","last_peered":"2024-03-11T22:57:32.475768+0000","last_clean":"2024-03-11T22:57:32.475768+0000","last_became_active":"2024-03-11T22:57:03.736315+0000","last_became_peered":"2024-03-11T22:57:03.736315+0000","last_unstale":"2024-03-11T22:57:32.475768+0000","last_undegraded":"2024-03-11T22:57:32.475768+0000","last_fullsized":"2024-03-11T22:57:32.475768+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:20:16.776864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476090+0000","last_change":"2024-03-11T22:57:05.739121+0000","last_active":"2024-03-11T22:57:32.476090+0000","last_peered":"2024-03-11T22:57:32.476090+0000","last_clean":"2024-03-11T22:57:32.476090+0000","last_became_active":"2024-03-11T22:57:05.738814+0000","last_became_peered":"2024-03-11T22:57:05.738814+0000","last_unstale":"2024-03-11T22:57:32.476090+0000","last_undegraded":"2024-03-11T22:57:32.476090+0000","last_fullsized":"2024-03-11T22:57:32.476090+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:19:20.271529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477249+0000","last_change":"2024-03-11T22:57:09.761298+0000","last_active":"2024-03-11T22:57:32.477249+0000","last_peered":"2024-03-11T22:57:32.477249+0000","last_clean":"2024-03-11T22:57:32.477249+0000","last_became_active":"2024-03-11T22:57:09.760246+0000","last_became_peered":"2024-03-11T22:57:09.760246+0000","last_unstale":"2024-03-11T22:57:32.477249+0000","last_undegraded":"2024-03-11T22:57:32.477249+0000","last_fullsized":"2024-03-11T22:57:32.477249+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:40:43.800496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.791093+0000","last_change":"2024-03-11T22:57:07.752518+0000","last_active":"2024-03-11T22:57:48.791093+0000","last_peered":"2024-03-11T22:57:48.791093+0000","last_clean":"2024-03-11T22:57:48.791093+0000","last_became_active":"2024-03-11T22:57:07.752045+0000","last_became_peered":"2024-03-11T22:57:07.752045+0000","last_unstale":"2024-03-11T22:57:48.791093+0000","last_undegraded":"2024-03-11T22:57:48.791093+0000","last_fullsized":"2024-03-11T22:57:48.791093+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:53:37.387595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474500+0000","last_change":"2024-03-11T22:57:03.739275+0000","last_active":"2024-03-11T22:57:32.474500+0000","last_peered":"2024-03-11T22:57:32.474500+0000","last_clean":"2024-03-11T22:57:32.474500+0000","last_became_active":"2024-03-11T22:57:03.738920+0000","last_became_peered":"2024-03-11T22:57:03.738920+0000","last_unstale":"2024-03-11T22:57:32.474500+0000","last_undegraded":"2024-03-11T22:57:32.474500+0000","last_fullsized":"2024-03-11T22:57:32.474500+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:56:38.532008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478608+0000","last_change":"2024-03-11T22:57:05.740777+0000","last_active":"2024-03-11T22:57:32.478608+0000","last_peered":"2024-03-11T22:57:32.478608+0000","last_clean":"2024-03-11T22:57:32.478608+0000","last_became_active":"2024-03-11T22:57:05.737954+0000","last_became_peered":"2024-03-11T22:57:05.737954+0000","last_unstale":"2024-03-11T22:57:32.478608+0000","last_undegraded":"2024-03-11T22:57:32.478608+0000","last_fullsized":"2024-03-11T22:57:32.478608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:45:58.403717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477332+0000","last_change":"2024-03-11T22:57:09.762975+0000","last_active":"2024-03-11T22:57:32.477332+0000","last_peered":"2024-03-11T22:57:32.477332+0000","last_clean":"2024-03-11T22:57:32.477332+0000","last_became_active":"2024-03-11T22:57:09.762282+0000","last_became_peered":"2024-03-11T22:57:09.762282+0000","last_unstale":"2024-03-11T22:57:32.477332+0000","last_undegraded":"2024-03-11T22:57:32.477332+0000","last_fullsized":"2024-03-11T22:57:32.477332+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:19:24.527788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477147+0000","last_change":"2024-03-11T22:57:07.749624+0000","last_active":"2024-03-11T22:57:32.477147+0000","last_peered":"2024-03-11T22:57:32.477147+0000","last_clean":"2024-03-11T22:57:32.477147+0000","last_became_active":"2024-03-11T22:57:07.749256+0000","last_became_peered":"2024-03-11T22:57:07.749256+0000","last_unstale":"2024-03-11T22:57:32.477147+0000","last_undegraded":"2024-03-11T22:57:32.477147+0000","last_fullsized":"2024-03-11T22:57:32.477147+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:53.679709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476614+0000","last_change":"2024-03-11T22:57:03.739904+0000","last_active":"2024-03-11T22:57:32.476614+0000","last_peered":"2024-03-11T22:57:32.476614+0000","last_clean":"2024-03-11T22:57:32.476614+0000","last_became_active":"2024-03-11T22:57:03.739444+0000","last_became_peered":"2024-03-11T22:57:03.739444+0000","last_unstale":"2024-03-11T22:57:32.476614+0000","last_undegraded":"2024-03-11T22:57:32.476614+0000","last_fullsized":"2024-03-11T22:57:32.476614+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:37.314228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476077+0000","last_change":"2024-03-11T22:57:05.739830+0000","last_active":"2024-03-11T22:57:32.476077+0000","last_peered":"2024-03-11T22:57:32.476077+0000","last_clean":"2024-03-11T22:57:32.476077+0000","last_became_active":"2024-03-11T22:57:05.739388+0000","last_became_peered":"2024-03-11T22:57:05.739388+0000","last_unstale":"2024-03-11T22:57:32.476077+0000","last_undegraded":"2024-03-11T22:57:32.476077+0000","last_fullsized":"2024-03-11T22:57:32.476077+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:06:02.238236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476848+0000","last_change":"2024-03-11T22:57:09.765113+0000","last_active":"2024-03-11T22:57:32.476848+0000","last_peered":"2024-03-11T22:57:32.476848+0000","last_clean":"2024-03-11T22:57:32.476848+0000","last_became_active":"2024-03-11T22:57:09.764533+0000","last_became_peered":"2024-03-11T22:57:09.764533+0000","last_unstale":"2024-03-11T22:57:32.476848+0000","last_undegraded":"2024-03-11T22:57:32.476848+0000","last_fullsized":"2024-03-11T22:57:32.476848+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:01:20.844411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476829+0000","last_change":"2024-03-11T22:57:07.753488+0000","last_active":"2024-03-11T22:57:32.476829+0000","last_peered":"2024-03-11T22:57:32.476829+0000","last_clean":"2024-03-11T22:57:32.476829+0000","last_became_active":"2024-03-11T22:57:07.752952+0000","last_became_peered":"2024-03-11T22:57:07.752952+0000","last_unstale":"2024-03-11T22:57:32.476829+0000","last_undegraded":"2024-03-11T22:57:32.476829+0000","last_fullsized":"2024-03-11T22:57:32.476829+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:46:04.219143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475038+0000","last_change":"2024-03-11T22:57:03.736790+0000","last_active":"2024-03-11T22:57:32.475038+0000","last_peered":"2024-03-11T22:57:32.475038+0000","last_clean":"2024-03-11T22:57:32.475038+0000","last_became_active":"2024-03-11T22:57:03.735880+0000","last_became_peered":"2024-03-11T22:57:03.735880+0000","last_unstale":"2024-03-11T22:57:32.475038+0000","last_undegraded":"2024-03-11T22:57:32.475038+0000","last_fullsized":"2024-03-11T22:57:32.475038+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:08:47.646016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474487+0000","last_change":"2024-03-11T22:57:05.739112+0000","last_active":"2024-03-11T22:57:32.474487+0000","last_peered":"2024-03-11T22:57:32.474487+0000","last_clean":"2024-03-11T22:57:32.474487+0000","last_became_active":"2024-03-11T22:57:05.738846+0000","last_became_peered":"2024-03-11T22:57:05.738846+0000","last_unstale":"2024-03-11T22:57:32.474487+0000","last_undegraded":"2024-03-11T22:57:32.474487+0000","last_fullsized":"2024-03-11T22:57:32.474487+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:05:14.460883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475266+0000","last_change":"2024-03-11T22:57:09.760083+0000","last_active":"2024-03-11T22:57:32.475266+0000","last_peered":"2024-03-11T22:57:32.475266+0000","last_clean":"2024-03-11T22:57:32.475266+0000","last_became_active":"2024-03-11T22:57:09.759794+0000","last_became_peered":"2024-03-11T22:57:09.759794+0000","last_unstale":"2024-03-11T22:57:32.475266+0000","last_undegraded":"2024-03-11T22:57:32.475266+0000","last_fullsized":"2024-03-11T22:57:32.475266+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:54:32.316324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476928+0000","last_change":"2024-03-11T22:57:03.736723+0000","last_active":"2024-03-11T22:57:32.476928+0000","last_peered":"2024-03-11T22:57:32.476928+0000","last_clean":"2024-03-11T22:57:32.476928+0000","last_became_active":"2024-03-11T22:57:03.736417+0000","last_became_peered":"2024-03-11T22:57:03.736417+0000","last_unstale":"2024-03-11T22:57:32.476928+0000","last_undegraded":"2024-03-11T22:57:32.476928+0000","last_fullsized":"2024-03-11T22:57:32.476928+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:19:51.144744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476872+0000","last_change":"2024-03-11T22:57:05.743444+0000","last_active":"2024-03-11T22:57:32.476872+0000","last_peered":"2024-03-11T22:57:32.476872+0000","last_clean":"2024-03-11T22:57:32.476872+0000","last_became_active":"2024-03-11T22:57:05.743208+0000","last_became_peered":"2024-03-11T22:57:05.743208+0000","last_unstale":"2024-03-11T22:57:32.476872+0000","last_undegraded":"2024-03-11T22:57:32.476872+0000","last_fullsized":"2024-03-11T22:57:32.476872+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:37:27.199004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.791080+0000","last_change":"2024-03-11T22:57:07.749811+0000","last_active":"2024-03-11T22:57:48.791080+0000","last_peered":"2024-03-11T22:57:48.791080+0000","last_clean":"2024-03-11T22:57:48.791080+0000","last_became_active":"2024-03-11T22:57:07.749356+0000","last_became_peered":"2024-03-11T22:57:07.749356+0000","last_unstale":"2024-03-11T22:57:48.791080+0000","last_undegraded":"2024-03-11T22:57:48.791080+0000","last_fullsized":"2024-03-11T22:57:48.791080+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:01:19.014709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476313+0000","last_change":"2024-03-11T22:57:09.763668+0000","last_active":"2024-03-11T22:57:32.476313+0000","last_peered":"2024-03-11T22:57:32.476313+0000","last_clean":"2024-03-11T22:57:32.476313+0000","last_became_active":"2024-03-11T22:57:09.763290+0000","last_became_peered":"2024-03-11T22:57:09.763290+0000","last_unstale":"2024-03-11T22:57:32.476313+0000","last_undegraded":"2024-03-11T22:57:32.476313+0000","last_fullsized":"2024-03-11T22:57:32.476313+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:08:00.259652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478584+0000","last_change":"2024-03-11T22:57:03.735290+0000","last_active":"2024-03-11T22:57:32.478584+0000","last_peered":"2024-03-11T22:57:32.478584+0000","last_clean":"2024-03-11T22:57:32.478584+0000","last_became_active":"2024-03-11T22:57:03.734975+0000","last_became_peered":"2024-03-11T22:57:03.734975+0000","last_unstale":"2024-03-11T22:57:32.478584+0000","last_undegraded":"2024-03-11T22:57:32.478584+0000","last_fullsized":"2024-03-11T22:57:32.478584+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:00:23.621572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.790921+0000","last_change":"2024-03-11T22:57:05.739866+0000","last_active":"2024-03-11T22:57:48.790921+0000","last_peered":"2024-03-11T22:57:48.790921+0000","last_clean":"2024-03-11T22:57:48.790921+0000","last_became_active":"2024-03-11T22:57:05.739378+0000","last_became_peered":"2024-03-11T22:57:05.739378+0000","last_unstale":"2024-03-11T22:57:48.790921+0000","last_undegraded":"2024-03-11T22:57:48.790921+0000","last_fullsized":"2024-03-11T22:57:48.790921+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:40:33.108757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474370+0000","last_change":"2024-03-11T22:57:07.751390+0000","last_active":"2024-03-11T22:57:32.474370+0000","last_peered":"2024-03-11T22:57:32.474370+0000","last_clean":"2024-03-11T22:57:32.474370+0000","last_became_active":"2024-03-11T22:57:07.751124+0000","last_became_peered":"2024-03-11T22:57:07.751124+0000","last_unstale":"2024-03-11T22:57:32.474370+0000","last_undegraded":"2024-03-11T22:57:32.474370+0000","last_fullsized":"2024-03-11T22:57:32.474370+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:47:10.506049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476336+0000","last_change":"2024-03-11T22:57:09.763967+0000","last_active":"2024-03-11T22:57:32.476336+0000","last_peered":"2024-03-11T22:57:32.476336+0000","last_clean":"2024-03-11T22:57:32.476336+0000","last_became_active":"2024-03-11T22:57:09.763447+0000","last_became_peered":"2024-03-11T22:57:09.763447+0000","last_unstale":"2024-03-11T22:57:32.476336+0000","last_undegraded":"2024-03-11T22:57:32.476336+0000","last_fullsized":"2024-03-11T22:57:32.476336+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:51:54.973911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476839+0000","last_change":"2024-03-11T22:57:03.735738+0000","last_active":"2024-03-11T22:57:32.476839+0000","last_peered":"2024-03-11T22:57:32.476839+0000","last_clean":"2024-03-11T22:57:32.476839+0000","last_became_active":"2024-03-11T22:57:03.735048+0000","last_became_peered":"2024-03-11T22:57:03.735048+0000","last_unstale":"2024-03-11T22:57:32.476839+0000","last_undegraded":"2024-03-11T22:57:32.476839+0000","last_fullsized":"2024-03-11T22:57:32.476839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:45:29.210506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'19","reported_seq":59,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478206+0000","last_change":"2024-03-11T22:57:05.741126+0000","last_active":"2024-03-11T22:57:32.478206+0000","last_peered":"2024-03-11T22:57:32.478206+0000","last_clean":"2024-03-11T22:57:32.478206+0000","last_became_active":"2024-03-11T22:57:05.739807+0000","last_became_peered":"2024-03-11T22:57:05.739807+0000","last_unstale":"2024-03-11T22:57:32.478206+0000","last_undegraded":"2024-03-11T22:57:32.478206+0000","last_fullsized":"2024-03-11T22:57:32.478206+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:41:21.142193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478199+0000","last_change":"2024-03-11T22:57:07.752439+0000","last_active":"2024-03-11T22:57:32.478199+0000","last_peered":"2024-03-11T22:57:32.478199+0000","last_clean":"2024-03-11T22:57:32.478199+0000","last_became_active":"2024-03-11T22:57:07.752092+0000","last_became_peered":"2024-03-11T22:57:07.752092+0000","last_unstale":"2024-03-11T22:57:32.478199+0000","last_undegraded":"2024-03-11T22:57:32.478199+0000","last_fullsized":"2024-03-11T22:57:32.478199+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:28:39.011327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477425+0000","last_change":"2024-03-11T22:57:09.764809+0000","last_active":"2024-03-11T22:57:32.477425+0000","last_peered":"2024-03-11T22:57:32.477425+0000","last_clean":"2024-03-11T22:57:32.477425+0000","last_became_active":"2024-03-11T22:57:09.764193+0000","last_became_peered":"2024-03-11T22:57:09.764193+0000","last_unstale":"2024-03-11T22:57:32.477425+0000","last_undegraded":"2024-03-11T22:57:32.477425+0000","last_fullsized":"2024-03-11T22:57:32.477425+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:50:53.719080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'18","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474002+0000","last_change":"2024-03-11T22:57:05.742991+0000","last_active":"2024-03-11T22:57:32.474002+0000","last_peered":"2024-03-11T22:57:32.474002+0000","last_clean":"2024-03-11T22:57:32.474002+0000","last_became_active":"2024-03-11T22:57:05.742609+0000","last_became_peered":"2024-03-11T22:57:05.742609+0000","last_unstale":"2024-03-11T22:57:32.474002+0000","last_undegraded":"2024-03-11T22:57:32.474002+0000","last_fullsized":"2024-03-11T22:57:32.474002+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:48:40.260948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475976+0000","last_change":"2024-03-11T22:57:03.736474+0000","last_active":"2024-03-11T22:57:32.475976+0000","last_peered":"2024-03-11T22:57:32.475976+0000","last_clean":"2024-03-11T22:57:32.475976+0000","last_became_active":"2024-03-11T22:57:03.735410+0000","last_became_peered":"2024-03-11T22:57:03.735410+0000","last_unstale":"2024-03-11T22:57:32.475976+0000","last_undegraded":"2024-03-11T22:57:32.475976+0000","last_fullsized":"2024-03-11T22:57:32.475976+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:29:49.633968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477515+0000","last_change":"2024-03-11T22:57:07.748819+0000","last_active":"2024-03-11T22:57:32.477515+0000","last_peered":"2024-03-11T22:57:32.477515+0000","last_clean":"2024-03-11T22:57:32.477515+0000","last_became_active":"2024-03-11T22:57:07.748526+0000","last_became_peered":"2024-03-11T22:57:07.748526+0000","last_unstale":"2024-03-11T22:57:32.477515+0000","last_undegraded":"2024-03-11T22:57:32.477515+0000","last_fullsized":"2024-03-11T22:57:32.477515+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:39:27.752699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475016+0000","last_change":"2024-03-11T22:57:09.759517+0000","last_active":"2024-03-11T22:57:32.475016+0000","last_peered":"2024-03-11T22:57:32.475016+0000","last_clean":"2024-03-11T22:57:32.475016+0000","last_became_active":"2024-03-11T22:57:09.759184+0000","last_became_peered":"2024-03-11T22:57:09.759184+0000","last_unstale":"2024-03-11T22:57:32.475016+0000","last_undegraded":"2024-03-11T22:57:32.475016+0000","last_fullsized":"2024-03-11T22:57:32.475016+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:41:47.749653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475874+0000","last_change":"2024-03-11T22:57:05.739628+0000","last_active":"2024-03-11T22:57:32.475874+0000","last_peered":"2024-03-11T22:57:32.475874+0000","last_clean":"2024-03-11T22:57:32.475874+0000","last_became_active":"2024-03-11T22:57:05.739302+0000","last_became_peered":"2024-03-11T22:57:05.739302+0000","last_unstale":"2024-03-11T22:57:32.475874+0000","last_undegraded":"2024-03-11T22:57:32.475874+0000","last_fullsized":"2024-03-11T22:57:32.475874+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:24:19.391447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477659+0000","last_change":"2024-03-11T22:57:03.740036+0000","last_active":"2024-03-11T22:57:32.477659+0000","last_peered":"2024-03-11T22:57:32.477659+0000","last_clean":"2024-03-11T22:57:32.477659+0000","last_became_active":"2024-03-11T22:57:03.739677+0000","last_became_peered":"2024-03-11T22:57:03.739677+0000","last_unstale":"2024-03-11T22:57:32.477659+0000","last_undegraded":"2024-03-11T22:57:32.477659+0000","last_fullsized":"2024-03-11T22:57:32.477659+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:07:19.119317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:34.579443+0000","last_change":"2024-03-11T22:56:50.610051+0000","last_active":"2024-03-11T22:57:34.579443+0000","last_peered":"2024-03-11T22:57:34.579443+0000","last_clean":"2024-03-11T22:57:34.579443+0000","last_became_active":"2024-03-11T22:56:50.609670+0000","last_became_peered":"2024-03-11T22:56:50.609670+0000","last_unstale":"2024-03-11T22:57:34.579443+0000","last_undegraded":"2024-03-11T22:57:34.579443+0000","last_fullsized":"2024-03-11T22:57:34.579443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:56:49.595444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:56:49.595444+0000","last_clean_scrub_stamp":"2024-03-11T22:56:49.595444+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:28:42.250456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478674+0000","last_change":"2024-03-11T22:57:07.755284+0000","last_active":"2024-03-11T22:57:32.478674+0000","last_peered":"2024-03-11T22:57:32.478674+0000","last_clean":"2024-03-11T22:57:32.478674+0000","last_became_active":"2024-03-11T22:57:07.754245+0000","last_became_peered":"2024-03-11T22:57:07.754245+0000","last_unstale":"2024-03-11T22:57:32.478674+0000","last_undegraded":"2024-03-11T22:57:32.478674+0000","last_fullsized":"2024-03-11T22:57:32.478674+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:17:29.004280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478663+0000","last_change":"2024-03-11T22:57:09.762845+0000","last_active":"2024-03-11T22:57:32.478663+0000","last_peered":"2024-03-11T22:57:32.478663+0000","last_clean":"2024-03-11T22:57:32.478663+0000","last_became_active":"2024-03-11T22:57:09.761497+0000","last_became_peered":"2024-03-11T22:57:09.761497+0000","last_unstale":"2024-03-11T22:57:32.478663+0000","last_undegraded":"2024-03-11T22:57:32.478663+0000","last_fullsized":"2024-03-11T22:57:32.478663+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:30:27.509349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'14","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476690+0000","last_change":"2024-03-11T22:57:05.740094+0000","last_active":"2024-03-11T22:57:32.476690+0000","last_peered":"2024-03-11T22:57:32.476690+0000","last_clean":"2024-03-11T22:57:32.476690+0000","last_became_active":"2024-03-11T22:57:05.739778+0000","last_became_peered":"2024-03-11T22:57:05.739778+0000","last_unstale":"2024-03-11T22:57:32.476690+0000","last_undegraded":"2024-03-11T22:57:32.476690+0000","last_fullsized":"2024-03-11T22:57:32.476690+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:54:45.526939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476100+0000","last_change":"2024-03-11T22:57:03.740222+0000","last_active":"2024-03-11T22:57:32.476100+0000","last_peered":"2024-03-11T22:57:32.476100+0000","last_clean":"2024-03-11T22:57:32.476100+0000","last_became_active":"2024-03-11T22:57:03.739551+0000","last_became_peered":"2024-03-11T22:57:03.739551+0000","last_unstale":"2024-03-11T22:57:32.476100+0000","last_undegraded":"2024-03-11T22:57:32.476100+0000","last_fullsized":"2024-03-11T22:57:32.476100+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:41:53.539081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475513+0000","last_change":"2024-03-11T22:57:07.751750+0000","last_active":"2024-03-11T22:57:32.475513+0000","last_peered":"2024-03-11T22:57:32.475513+0000","last_clean":"2024-03-11T22:57:32.475513+0000","last_became_active":"2024-03-11T22:57:07.751405+0000","last_became_peered":"2024-03-11T22:57:07.751405+0000","last_unstale":"2024-03-11T22:57:32.475513+0000","last_undegraded":"2024-03-11T22:57:32.475513+0000","last_fullsized":"2024-03-11T22:57:32.475513+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:10:02.426990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476683+0000","last_change":"2024-03-11T22:57:09.762402+0000","last_active":"2024-03-11T22:57:32.476683+0000","last_peered":"2024-03-11T22:57:32.476683+0000","last_clean":"2024-03-11T22:57:32.476683+0000","last_became_active":"2024-03-11T22:57:09.761136+0000","last_became_peered":"2024-03-11T22:57:09.761136+0000","last_unstale":"2024-03-11T22:57:32.476683+0000","last_undegraded":"2024-03-11T22:57:32.476683+0000","last_fullsized":"2024-03-11T22:57:32.476683+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:42:24.150646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475987+0000","last_change":"2024-03-11T22:57:03.738080+0000","last_active":"2024-03-11T22:57:32.475987+0000","last_peered":"2024-03-11T22:57:32.475987+0000","last_clean":"2024-03-11T22:57:32.475987+0000","last_became_active":"2024-03-11T22:57:03.737766+0000","last_became_peered":"2024-03-11T22:57:03.737766+0000","last_unstale":"2024-03-11T22:57:32.475987+0000","last_undegraded":"2024-03-11T22:57:32.475987+0000","last_fullsized":"2024-03-11T22:57:32.475987+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:38:40.674857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'28","reported_seq":82,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.790770+0000","last_change":"2024-03-11T22:57:05.745716+0000","last_active":"2024-03-11T22:57:48.790770+0000","last_peered":"2024-03-11T22:57:48.790770+0000","last_clean":"2024-03-11T22:57:48.790770+0000","last_became_active":"2024-03-11T22:57:05.745385+0000","last_became_peered":"2024-03-11T22:57:05.745385+0000","last_unstale":"2024-03-11T22:57:48.790770+0000","last_undegraded":"2024-03-11T22:57:48.790770+0000","last_fullsized":"2024-03-11T22:57:48.790770+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:25:03.189221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.791044+0000","last_change":"2024-03-11T22:57:07.748406+0000","last_active":"2024-03-11T22:57:48.791044+0000","last_peered":"2024-03-11T22:57:48.791044+0000","last_clean":"2024-03-11T22:57:48.791044+0000","last_became_active":"2024-03-11T22:57:07.748118+0000","last_became_peered":"2024-03-11T22:57:07.748118+0000","last_unstale":"2024-03-11T22:57:48.791044+0000","last_undegraded":"2024-03-11T22:57:48.791044+0000","last_fullsized":"2024-03-11T22:57:48.791044+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:08:14.025989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476133+0000","last_change":"2024-03-11T22:57:09.760828+0000","last_active":"2024-03-11T22:57:32.476133+0000","last_peered":"2024-03-11T22:57:32.476133+0000","last_clean":"2024-03-11T22:57:32.476133+0000","last_became_active":"2024-03-11T22:57:09.760424+0000","last_became_peered":"2024-03-11T22:57:09.760424+0000","last_unstale":"2024-03-11T22:57:32.476133+0000","last_undegraded":"2024-03-11T22:57:32.476133+0000","last_fullsized":"2024-03-11T22:57:32.476133+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:23:32.505783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476926+0000","last_change":"2024-03-11T22:57:03.736375+0000","last_active":"2024-03-11T22:57:32.476926+0000","last_peered":"2024-03-11T22:57:32.476926+0000","last_clean":"2024-03-11T22:57:32.476926+0000","last_became_active":"2024-03-11T22:57:03.736083+0000","last_became_peered":"2024-03-11T22:57:03.736083+0000","last_unstale":"2024-03-11T22:57:32.476926+0000","last_undegraded":"2024-03-11T22:57:32.476926+0000","last_fullsized":"2024-03-11T22:57:32.476926+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:10:18.815045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'13","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478233+0000","last_change":"2024-03-11T22:57:05.738926+0000","last_active":"2024-03-11T22:57:32.478233+0000","last_peered":"2024-03-11T22:57:32.478233+0000","last_clean":"2024-03-11T22:57:32.478233+0000","last_became_active":"2024-03-11T22:57:05.738589+0000","last_became_peered":"2024-03-11T22:57:05.738589+0000","last_unstale":"2024-03-11T22:57:32.478233+0000","last_undegraded":"2024-03-11T22:57:32.478233+0000","last_fullsized":"2024-03-11T22:57:32.478233+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:25:32.229745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.790765+0000","last_change":"2024-03-11T22:57:07.753158+0000","last_active":"2024-03-11T22:57:48.790765+0000","last_peered":"2024-03-11T22:57:48.790765+0000","last_clean":"2024-03-11T22:57:48.790765+0000","last_became_active":"2024-03-11T22:57:07.752882+0000","last_became_peered":"2024-03-11T22:57:07.752882+0000","last_unstale":"2024-03-11T22:57:48.790765+0000","last_undegraded":"2024-03-11T22:57:48.790765+0000","last_fullsized":"2024-03-11T22:57:48.790765+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:46:39.603552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475237+0000","last_change":"2024-03-11T22:57:09.760445+0000","last_active":"2024-03-11T22:57:32.475237+0000","last_peered":"2024-03-11T22:57:32.475237+0000","last_clean":"2024-03-11T22:57:32.475237+0000","last_became_active":"2024-03-11T22:57:09.760136+0000","last_became_peered":"2024-03-11T22:57:09.760136+0000","last_unstale":"2024-03-11T22:57:32.475237+0000","last_undegraded":"2024-03-11T22:57:32.475237+0000","last_fullsized":"2024-03-11T22:57:32.475237+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T07:02:27.534194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.479039+0000","last_change":"2024-03-11T22:57:03.737827+0000","last_active":"2024-03-11T22:57:32.479039+0000","last_peered":"2024-03-11T22:57:32.479039+0000","last_clean":"2024-03-11T22:57:32.479039+0000","last_became_active":"2024-03-11T22:57:03.737544+0000","last_became_peered":"2024-03-11T22:57:03.737544+0000","last_unstale":"2024-03-11T22:57:32.479039+0000","last_undegraded":"2024-03-11T22:57:32.479039+0000","last_fullsized":"2024-03-11T22:57:32.479039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:18:32.149946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477134+0000","last_change":"2024-03-11T22:57:05.745719+0000","last_active":"2024-03-11T22:57:32.477134+0000","last_peered":"2024-03-11T22:57:32.477134+0000","last_clean":"2024-03-11T22:57:32.477134+0000","last_became_active":"2024-03-11T22:57:05.745306+0000","last_became_peered":"2024-03-11T22:57:05.745306+0000","last_unstale":"2024-03-11T22:57:32.477134+0000","last_undegraded":"2024-03-11T22:57:32.477134+0000","last_fullsized":"2024-03-11T22:57:32.477134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:48:18.544997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474181+0000","last_change":"2024-03-11T22:57:07.748519+0000","last_active":"2024-03-11T22:57:32.474181+0000","last_peered":"2024-03-11T22:57:32.474181+0000","last_clean":"2024-03-11T22:57:32.474181+0000","last_became_active":"2024-03-11T22:57:07.748216+0000","last_became_peered":"2024-03-11T22:57:07.748216+0000","last_unstale":"2024-03-11T22:57:32.474181+0000","last_undegraded":"2024-03-11T22:57:32.474181+0000","last_fullsized":"2024-03-11T22:57:32.474181+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:55:39.913218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478774+0000","last_change":"2024-03-11T22:57:09.763248+0000","last_active":"2024-03-11T22:57:32.478774+0000","last_peered":"2024-03-11T22:57:32.478774+0000","last_clean":"2024-03-11T22:57:32.478774+0000","last_became_active":"2024-03-11T22:57:09.762651+0000","last_became_peered":"2024-03-11T22:57:09.762651+0000","last_unstale":"2024-03-11T22:57:32.478774+0000","last_undegraded":"2024-03-11T22:57:32.478774+0000","last_fullsized":"2024-03-11T22:57:32.478774+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:25:00.302836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.790947+0000","last_change":"2024-03-11T22:57:07.752478+0000","last_active":"2024-03-11T22:57:48.790947+0000","last_peered":"2024-03-11T22:57:48.790947+0000","last_clean":"2024-03-11T22:57:48.790947+0000","last_became_active":"2024-03-11T22:57:07.751962+0000","last_became_peered":"2024-03-11T22:57:07.751962+0000","last_unstale":"2024-03-11T22:57:48.790947+0000","last_undegraded":"2024-03-11T22:57:48.790947+0000","last_fullsized":"2024-03-11T22:57:48.790947+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:40:32.348228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476268+0000","last_change":"2024-03-11T22:57:03.735624+0000","last_active":"2024-03-11T22:57:32.476268+0000","last_peered":"2024-03-11T22:57:32.476268+0000","last_clean":"2024-03-11T22:57:32.476268+0000","last_became_active":"2024-03-11T22:57:03.734510+0000","last_became_peered":"2024-03-11T22:57:03.734510+0000","last_unstale":"2024-03-11T22:57:32.476268+0000","last_undegraded":"2024-03-11T22:57:32.476268+0000","last_fullsized":"2024-03-11T22:57:32.476268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:15:30.311418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'12","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474812+0000","last_change":"2024-03-11T22:57:05.738370+0000","last_active":"2024-03-11T22:57:32.474812+0000","last_peered":"2024-03-11T22:57:32.474812+0000","last_clean":"2024-03-11T22:57:32.474812+0000","last_became_active":"2024-03-11T22:57:05.738062+0000","last_became_peered":"2024-03-11T22:57:05.738062+0000","last_unstale":"2024-03-11T22:57:32.474812+0000","last_undegraded":"2024-03-11T22:57:32.474812+0000","last_fullsized":"2024-03-11T22:57:32.474812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:00:56.868200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475929+0000","last_change":"2024-03-11T22:57:09.760014+0000","last_active":"2024-03-11T22:57:32.475929+0000","last_peered":"2024-03-11T22:57:32.475929+0000","last_clean":"2024-03-11T22:57:32.475929+0000","last_became_active":"2024-03-11T22:57:09.759727+0000","last_became_peered":"2024-03-11T22:57:09.759727+0000","last_unstale":"2024-03-11T22:57:32.475929+0000","last_undegraded":"2024-03-11T22:57:32.475929+0000","last_fullsized":"2024-03-11T22:57:32.475929+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:56:05.619543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477190+0000","last_change":"2024-03-11T22:57:07.749161+0000","last_active":"2024-03-11T22:57:32.477190+0000","last_peered":"2024-03-11T22:57:32.477190+0000","last_clean":"2024-03-11T22:57:32.477190+0000","last_became_active":"2024-03-11T22:57:07.748939+0000","last_became_peered":"2024-03-11T22:57:07.748939+0000","last_unstale":"2024-03-11T22:57:32.477190+0000","last_undegraded":"2024-03-11T22:57:32.477190+0000","last_fullsized":"2024-03-11T22:57:32.477190+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:45:55.683829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476516+0000","last_change":"2024-03-11T22:57:03.736236+0000","last_active":"2024-03-11T22:57:32.476516+0000","last_peered":"2024-03-11T22:57:32.476516+0000","last_clean":"2024-03-11T22:57:32.476516+0000","last_became_active":"2024-03-11T22:57:03.735432+0000","last_became_peered":"2024-03-11T22:57:03.735432+0000","last_unstale":"2024-03-11T22:57:32.476516+0000","last_undegraded":"2024-03-11T22:57:32.476516+0000","last_fullsized":"2024-03-11T22:57:32.476516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:34:16.402095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474978+0000","last_change":"2024-03-11T22:57:05.743022+0000","last_active":"2024-03-11T22:57:32.474978+0000","last_peered":"2024-03-11T22:57:32.474978+0000","last_clean":"2024-03-11T22:57:32.474978+0000","last_became_active":"2024-03-11T22:57:05.742613+0000","last_became_peered":"2024-03-11T22:57:05.742613+0000","last_unstale":"2024-03-11T22:57:32.474978+0000","last_undegraded":"2024-03-11T22:57:32.474978+0000","last_fullsized":"2024-03-11T22:57:32.474978+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:56:10.012331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475276+0000","last_change":"2024-03-11T22:57:09.760368+0000","last_active":"2024-03-11T22:57:32.475276+0000","last_peered":"2024-03-11T22:57:32.475276+0000","last_clean":"2024-03-11T22:57:32.475276+0000","last_became_active":"2024-03-11T22:57:09.759191+0000","last_became_peered":"2024-03-11T22:57:09.759191+0000","last_unstale":"2024-03-11T22:57:32.475276+0000","last_undegraded":"2024-03-11T22:57:32.475276+0000","last_fullsized":"2024-03-11T22:57:32.475276+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:58:01.807812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.474701+0000","last_change":"2024-03-11T22:57:03.739018+0000","last_active":"2024-03-11T22:57:32.474701+0000","last_peered":"2024-03-11T22:57:32.474701+0000","last_clean":"2024-03-11T22:57:32.474701+0000","last_became_active":"2024-03-11T22:57:03.738314+0000","last_became_peered":"2024-03-11T22:57:03.738314+0000","last_unstale":"2024-03-11T22:57:32.474701+0000","last_undegraded":"2024-03-11T22:57:32.474701+0000","last_fullsized":"2024-03-11T22:57:32.474701+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:41:28.323306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477022+0000","last_change":"2024-03-11T22:57:05.736367+0000","last_active":"2024-03-11T22:57:32.477022+0000","last_peered":"2024-03-11T22:57:32.477022+0000","last_clean":"2024-03-11T22:57:32.477022+0000","last_became_active":"2024-03-11T22:57:05.736118+0000","last_became_peered":"2024-03-11T22:57:05.736118+0000","last_unstale":"2024-03-11T22:57:32.477022+0000","last_undegraded":"2024-03-11T22:57:32.477022+0000","last_fullsized":"2024-03-11T22:57:32.477022+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:09:55.425498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477548+0000","last_change":"2024-03-11T22:57:07.754093+0000","last_active":"2024-03-11T22:57:32.477548+0000","last_peered":"2024-03-11T22:57:32.477548+0000","last_clean":"2024-03-11T22:57:32.477548+0000","last_became_active":"2024-03-11T22:57:07.753396+0000","last_became_peered":"2024-03-11T22:57:07.753396+0000","last_unstale":"2024-03-11T22:57:32.477548+0000","last_undegraded":"2024-03-11T22:57:32.477548+0000","last_fullsized":"2024-03-11T22:57:32.477548+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:17:11.825147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477467+0000","last_change":"2024-03-11T22:57:09.762171+0000","last_active":"2024-03-11T22:57:32.477467+0000","last_peered":"2024-03-11T22:57:32.477467+0000","last_clean":"2024-03-11T22:57:32.477467+0000","last_became_active":"2024-03-11T22:57:09.761772+0000","last_became_peered":"2024-03-11T22:57:09.761772+0000","last_unstale":"2024-03-11T22:57:32.477467+0000","last_undegraded":"2024-03-11T22:57:32.477467+0000","last_fullsized":"2024-03-11T22:57:32.477467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T03:39:31.437327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:48.790912+0000","last_change":"2024-03-11T22:57:07.753577+0000","last_active":"2024-03-11T22:57:48.790912+0000","last_peered":"2024-03-11T22:57:48.790912+0000","last_clean":"2024-03-11T22:57:48.790912+0000","last_became_active":"2024-03-11T22:57:07.752453+0000","last_became_peered":"2024-03-11T22:57:07.752453+0000","last_unstale":"2024-03-11T22:57:48.790912+0000","last_undegraded":"2024-03-11T22:57:48.790912+0000","last_fullsized":"2024-03-11T22:57:48.790912+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:28:10.365849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476075+0000","last_change":"2024-03-11T22:57:03.737879+0000","last_active":"2024-03-11T22:57:32.476075+0000","last_peered":"2024-03-11T22:57:32.476075+0000","last_clean":"2024-03-11T22:57:32.476075+0000","last_became_active":"2024-03-11T22:57:03.737600+0000","last_became_peered":"2024-03-11T22:57:03.737600+0000","last_unstale":"2024-03-11T22:57:32.476075+0000","last_undegraded":"2024-03-11T22:57:32.476075+0000","last_fullsized":"2024-03-11T22:57:32.476075+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T09:27:23.936281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'19","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476954+0000","last_change":"2024-03-11T22:57:05.741186+0000","last_active":"2024-03-11T22:57:32.476954+0000","last_peered":"2024-03-11T22:57:32.476954+0000","last_clean":"2024-03-11T22:57:32.476954+0000","last_became_active":"2024-03-11T22:57:05.740675+0000","last_became_peered":"2024-03-11T22:57:05.740675+0000","last_unstale":"2024-03-11T22:57:32.476954+0000","last_undegraded":"2024-03-11T22:57:32.476954+0000","last_fullsized":"2024-03-11T22:57:32.476954+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:25:50.580476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475645+0000","last_change":"2024-03-11T22:57:09.760483+0000","last_active":"2024-03-11T22:57:32.475645+0000","last_peered":"2024-03-11T22:57:32.475645+0000","last_clean":"2024-03-11T22:57:32.475645+0000","last_became_active":"2024-03-11T22:57:09.760172+0000","last_became_peered":"2024-03-11T22:57:09.760172+0000","last_unstale":"2024-03-11T22:57:32.475645+0000","last_undegraded":"2024-03-11T22:57:32.475645+0000","last_fullsized":"2024-03-11T22:57:32.475645+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:06:05.343154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475727+0000","last_change":"2024-03-11T22:57:03.736773+0000","last_active":"2024-03-11T22:57:32.475727+0000","last_peered":"2024-03-11T22:57:32.475727+0000","last_clean":"2024-03-11T22:57:32.475727+0000","last_became_active":"2024-03-11T22:57:03.736443+0000","last_became_peered":"2024-03-11T22:57:03.736443+0000","last_unstale":"2024-03-11T22:57:32.475727+0000","last_undegraded":"2024-03-11T22:57:32.475727+0000","last_fullsized":"2024-03-11T22:57:32.475727+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:08:42.432197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478674+0000","last_change":"2024-03-11T22:57:05.742940+0000","last_active":"2024-03-11T22:57:32.478674+0000","last_peered":"2024-03-11T22:57:32.478674+0000","last_clean":"2024-03-11T22:57:32.478674+0000","last_became_active":"2024-03-11T22:57:05.741635+0000","last_became_peered":"2024-03-11T22:57:05.741635+0000","last_unstale":"2024-03-11T22:57:32.478674+0000","last_undegraded":"2024-03-11T22:57:32.478674+0000","last_fullsized":"2024-03-11T22:57:32.478674+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:04:45.518700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476651+0000","last_change":"2024-03-11T22:57:07.752540+0000","last_active":"2024-03-11T22:57:32.476651+0000","last_peered":"2024-03-11T22:57:32.476651+0000","last_clean":"2024-03-11T22:57:32.476651+0000","last_became_active":"2024-03-11T22:57:07.752124+0000","last_became_peered":"2024-03-11T22:57:07.752124+0000","last_unstale":"2024-03-11T22:57:32.476651+0000","last_undegraded":"2024-03-11T22:57:32.476651+0000","last_fullsized":"2024-03-11T22:57:32.476651+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T00:37:08.935596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475570+0000","last_change":"2024-03-11T22:57:09.763961+0000","last_active":"2024-03-11T22:57:32.475570+0000","last_peered":"2024-03-11T22:57:32.475570+0000","last_clean":"2024-03-11T22:57:32.475570+0000","last_became_active":"2024-03-11T22:57:09.763611+0000","last_became_peered":"2024-03-11T22:57:09.763611+0000","last_unstale":"2024-03-11T22:57:32.475570+0000","last_undegraded":"2024-03-11T22:57:32.475570+0000","last_fullsized":"2024-03-11T22:57:32.475570+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-12T23:51:06.959552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475674+0000","last_change":"2024-03-11T22:57:03.738159+0000","last_active":"2024-03-11T22:57:32.475674+0000","last_peered":"2024-03-11T22:57:32.475674+0000","last_clean":"2024-03-11T22:57:32.475674+0000","last_became_active":"2024-03-11T22:57:03.737871+0000","last_became_peered":"2024-03-11T22:57:03.737871+0000","last_unstale":"2024-03-11T22:57:32.475674+0000","last_undegraded":"2024-03-11T22:57:32.475674+0000","last_fullsized":"2024-03-11T22:57:32.475674+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T05:48:00.204054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.478031+0000","last_change":"2024-03-11T22:57:05.741721+0000","last_active":"2024-03-11T22:57:32.478031+0000","last_peered":"2024-03-11T22:57:32.478031+0000","last_clean":"2024-03-11T22:57:32.478031+0000","last_became_active":"2024-03-11T22:57:05.737966+0000","last_became_peered":"2024-03-11T22:57:05.737966+0000","last_unstale":"2024-03-11T22:57:32.478031+0000","last_undegraded":"2024-03-11T22:57:32.478031+0000","last_fullsized":"2024-03-11T22:57:32.478031+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:55:01.743914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476448+0000","last_change":"2024-03-11T22:57:07.748770+0000","last_active":"2024-03-11T22:57:32.476448+0000","last_peered":"2024-03-11T22:57:32.476448+0000","last_clean":"2024-03-11T22:57:32.476448+0000","last_became_active":"2024-03-11T22:57:07.748469+0000","last_became_peered":"2024-03-11T22:57:07.748469+0000","last_unstale":"2024-03-11T22:57:32.476448+0000","last_undegraded":"2024-03-11T22:57:32.476448+0000","last_fullsized":"2024-03-11T22:57:32.476448+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T10:38:04.628377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.476223+0000","last_change":"2024-03-11T22:57:09.760430+0000","last_active":"2024-03-11T22:57:32.476223+0000","last_peered":"2024-03-11T22:57:32.476223+0000","last_clean":"2024-03-11T22:57:32.476223+0000","last_became_active":"2024-03-11T22:57:09.759811+0000","last_became_peered":"2024-03-11T22:57:09.759811+0000","last_unstale":"2024-03-11T22:57:32.476223+0000","last_undegraded":"2024-03-11T22:57:32.476223+0000","last_fullsized":"2024-03-11T22:57:32.476223+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:08.737988+0000","last_clean_scrub_stamp":"2024-03-11T22:57:08.737988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T08:54:10.600615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.477039+0000","last_change":"2024-03-11T22:57:03.736375+0000","last_active":"2024-03-11T22:57:32.477039+0000","last_peered":"2024-03-11T22:57:32.477039+0000","last_clean":"2024-03-11T22:57:32.477039+0000","last_became_active":"2024-03-11T22:57:03.736116+0000","last_became_peered":"2024-03-11T22:57:03.736116+0000","last_unstale":"2024-03-11T22:57:32.477039+0000","last_undegraded":"2024-03-11T22:57:32.477039+0000","last_fullsized":"2024-03-11T22:57:32.477039+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:02.717348+0000","last_clean_scrub_stamp":"2024-03-11T22:57:02.717348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T02:54:59.226299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.479114+0000","last_change":"2024-03-11T22:57:05.743665+0000","last_active":"2024-03-11T22:57:32.479114+0000","last_peered":"2024-03-11T22:57:32.479114+0000","last_clean":"2024-03-11T22:57:32.479114+0000","last_became_active":"2024-03-11T22:57:05.743153+0000","last_became_peered":"2024-03-11T22:57:05.743153+0000","last_unstale":"2024-03-11T22:57:32.479114+0000","last_undegraded":"2024-03-11T22:57:32.479114+0000","last_fullsized":"2024-03-11T22:57:32.479114+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:04.723065+0000","last_clean_scrub_stamp":"2024-03-11T22:57:04.723065+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:59:00.702827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-03-11T22:57:32.475139+0000","last_change":"2024-03-11T22:57:07.752448+0000","last_active":"2024-03-11T22:57:32.475139+0000","last_peered":"2024-03-11T22:57:32.475139+0000","last_clean":"2024-03-11T22:57:32.475139+0000","last_became_active":"2024-03-11T22:57:07.752162+0000","last_became_peered":"2024-03-11T22:57:07.752162+0000","last_unstale":"2024-03-11T22:57:32.475139+0000","last_undegraded":"2024-03-11T22:57:32.475139+0000","last_fullsized":"2024-03-11T22:57:32.475139+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T22:57:06.728882+0000","last_clean_scrub_stamp":"2024-03-11T22:57:06.728882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T01:34:38.397636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411345,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704692,"statfs":{"total":95990841344,"available":95953604608,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738395,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32204,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708852,"statfs":{"total":95990841344,"available":95957864448,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T22:58:01.971 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T22:58:01.971 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T22:58:01.971 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T22:58:01.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T22:58:02.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:02.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:58:02.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:01 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:02.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:58:02.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:01 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:01 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:02.772 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:03.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:02 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:03.098 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:02 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:02 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.099 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:02 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:02 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:03.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:02 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:02 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:02 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:02 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:03.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:02 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:02 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:03.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:02 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:04.150 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:03 smithi107 ceph-mon[25849]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:04.151 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:03 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:58:03.994Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00288649s 2024-03-11T22:58:04.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:03 smithi131 ceph-mon[27544]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:04.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:03 smithi195 ceph-mon[27640]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:04.624 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-03-11T22:58:04.624 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T22:58:05.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:04 smithi107 ceph-mon[25849]: from='client.? 172.21.15.107:0/726797197' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:58:05.280 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T22:58:05.281 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T22:58:05.281 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-11T22:58:05.291 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi107.front.sepia.ceph.com 2024-03-11T22:58:05.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-03-11T22:58:05.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:04 smithi131 ceph-mon[27544]: from='client.? 172.21.15.107:0/726797197' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:58:05.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:04 smithi195 ceph-mon[27640]: from='client.? 172.21.15.107:0/726797197' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T22:58:05.534 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:06.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:05 smithi131 ceph-mon[27544]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:06.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:05 smithi195 ceph-mon[27640]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:06.362 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:05 smithi107 ceph-mon[25849]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:07.133 INFO:teuthology.orchestra.run.smithi107.stdout:167 167 2024-03-11T22:58:07.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch status' 2024-03-11T22:58:08.158 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:08.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:07 smithi107 ceph-mon[25849]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:08.271 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:07 smithi131 ceph-mon[27544]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:08.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:07 smithi195 ceph-mon[27640]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:09.240 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 systemd[1]: Starting Ceph grafana.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:09.550 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 podman[38799]: 2024-03-11 22:58:09.235985234 +0000 UTC m=+0.103924146 container create c5739fa709700d2b7f3f27bd753c5b6d1ea92c3d72b050909bf72f7a98bdad2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , architecture=x86_64, description=Ceph Grafana Container, distribution-scope=public, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, vcs-type=git, release=239.1651231664, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-11T22:58:09.550 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 podman[38799]: 2024-03-11 22:58:09.148489181 +0000 UTC m=+0.016428094 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T22:58:09.550 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 podman[38799]: 2024-03-11 22:58:09.4716593 +0000 UTC m=+0.339598213 container init c5739fa709700d2b7f3f27bd753c5b6d1ea92c3d72b050909bf72f7a98bdad2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, vcs-type=git, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, architecture=x86_64, description=Ceph Grafana Container, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T22:58:09.551 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 podman[38799]: 2024-03-11 22:58:09.477358552 +0000 UTC m=+0.345297469 container start c5739fa709700d2b7f3f27bd753c5b6d1ea92c3d72b050909bf72f7a98bdad2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, distribution-scope=public, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.buildah.version=1.32.0, release=239.1651231664, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-11T22:58:09.551 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 bash[38799]: c5739fa709700d2b7f3f27bd753c5b6d1ea92c3d72b050909bf72f7a98bdad2f 2024-03-11T22:58:09.801 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 systemd[1]: Started Ceph grafana.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:09.801 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.630806118Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T22:58:09.801 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.63116607Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T22:58:09.801 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.63118638Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631193392Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631199209Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631206073Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631212003Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631217629Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631223364Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631229009Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631234318Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631240422Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631245939Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=settings t=2024-03-11T22:58:09.631251624Z level=info msg="App mode production" 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=sqlstore t=2024-03-11T22:58:09.631294649Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=sqlstore t=2024-03-11T22:58:09.631311421Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T22:58:09.802 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.632194321Z level=info msg="Starting DB migrations" 2024-03-11T22:58:09.803 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.632272349Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.632795492Z level=info msg="Executing migration" id="create user table" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.633293988Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.634093718Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.634977499Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.635807103Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.636654286Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.637528236Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.638401318Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.639246926Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.640145097Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.640944932Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T22:58:09.804 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.641799155Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.642634052Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.642945478Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.645794941Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.648497558Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.651163708Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.651614948Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.654478797Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.657210854Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.660031523Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.662777172Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.665562257Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.668353772Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.671079987Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.674031369Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T22:58:09.805 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.676661353Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.679337149Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.682091501Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.685003631Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.687498262Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.6902403Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.69292071Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.69571662Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.698085506Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.70082864Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.703472299Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.705983543Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.70631448Z level=info msg="Executing migration" id="create star table" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.709020004Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T22:58:09.806 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.70943256Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.711923105Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.714561399Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.717198204Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.719770946Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.722371121Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.723064007Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.725342951Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.72610284Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.726938801Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.727347497Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.728162424Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.729118884Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.729792427Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.730216803Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T22:58:09.807 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.731008741Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.731871242Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.732660837Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.73354027Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.73438625Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.735045596Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.735945045Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.736439928Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.737231657Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.737774682Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.738297365Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.739098018Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.73963264Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.740463903Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.741300642Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T22:58:09.808 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.742162385Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.742482792Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.743320367Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.744170461Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.74500896Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.745818215Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.746856425Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.747517681Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.748416784Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.748853803Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.749293384Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.750220788Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.751344084Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.754069551Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.75449185Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T22:58:09.809 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.757286564Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.760019603Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.762552661Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.763118748Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.765885357Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.768688655Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.771437106Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.771843737Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.774657441Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.777603546Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.780392602Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.780814278Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.783999758Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.786291572Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T22:58:09.810 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.789185085Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T22:58:09.811 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.789618475Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T22:58:09.811 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.792331522Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T22:58:09.811 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.795056379Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T22:58:09.811 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.797692892Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T22:58:09.811 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.79844554Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T22:58:09.951 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:09 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ::ffff:172.21.15.131 - - [11/Mar/2024:22:58:09] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:58:10.053 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-mon[27544]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:10.053 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.053 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.053 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.053 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.800971873Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.803696559Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.806224263Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.808041176Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.810603675Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.813338457Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.816147325Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.816500093Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.819088442Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T22:58:10.054 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.819549812Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.821752837Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.824460106Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.824931785Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.825722981Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.826528621Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.827346077Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.827828085Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.828954025Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.829378115Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.82978286Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.830222421Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.830661108Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.830982793Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T22:58:10.055 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.831439063Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.831724867Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.832299408Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.832856135Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.833204588Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.833735659Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.834363065Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.834779354Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.835191297Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.835609465Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.836062414Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.836510281Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.836945867Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.837266708Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.837546081Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T22:58:10.056 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.838126275Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.838697524Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.839041837Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.839466033Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.839931625Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.840260077Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.840684257Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.841572026Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.842173818Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.842497455Z level=info msg="Executing migration" id="create session table" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.842948567Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.843268752Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.843565737Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.843984253Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T22:58:10.057 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.844392198Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T22:58:10.058 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.844675618Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T22:58:10.058 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.844976872Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T22:58:10.058 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.845296988Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T22:58:10.058 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.845607314Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T22:58:10.058 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.846016497Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T22:58:10.059 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.846333747Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T22:58:10.059 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.846910926Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T22:58:10.059 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.847242374Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T22:58:10.059 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.847811888Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T22:58:10.059 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.848442909Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T22:58:10.059 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.848759568Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.849211259Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.849680635Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.850192188Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.852026273Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.852809466Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.85325105Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.853644351Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.854094406Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.854517129Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.855971524Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.856373983Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.85680517Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.857157335Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T22:58:10.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.857535788Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.857916845Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.858551672Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.859199034Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.859856876Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.860564236Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.86099332Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.861330058Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.861636816Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.862050454Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.862493067Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.862857453Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.863299305Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.864741494Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T22:58:10.061 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.865461815Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.866125995Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.866976105Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.867433216Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.868248134Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.86893891Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.869254236Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.871351323Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.871813292Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.872144044Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.875068088Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.877782363Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.880454599Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.880929204Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T22:58:10.062 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.883581468Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.886084628Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.88897325Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.889720253Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.892467425Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.895300133Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.897956553Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.898414191Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.901119938Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.90270388Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.903816692Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.906610273Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.906982095Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.909635321Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.91208984Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T22:58:10.063 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.91471072Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.915460539Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.917923794Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.920677362Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.923150693Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.925486562Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.928211664Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.930782411Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.933380457Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.933835858Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.936472305Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.939211426Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.94191749Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.944386929Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T22:58:10.064 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.945155342Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.945992347Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.946452266Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.946866878Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.947722161Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.948573516Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.949304686Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.950142129Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.95098217Z level=info msg="Executing migration" id="create team table" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.951779112Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.952566041Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.953343027Z level=info msg="Executing migration" id="create team member table" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.954129849Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.954943027Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.96001245Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.960491457Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T22:58:10.065 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.961275586Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.962132885Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.96296225Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.963845211Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.964597347Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.965412922Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.965868355Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.966712119Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.967520543Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.967955605Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.968406512Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.969202298Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.969521759Z level=info msg="Executing migration" id="create tag table" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.970061709Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T22:58:10.066 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.970854248Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.971249566Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.972748905Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.973260489Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.975399531Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.97578693Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.976649191Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.977493779Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.978340242Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.979128049Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.979990774Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.980958101Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.981731649Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.982583508Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.983437492Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.984298738Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T22:58:10.067 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.985201294Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.986067862Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.987001677Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.9877141Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.98856145Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.9895483Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.990280237Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.993029147Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.9957946Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:09 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:09.998549452Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.001365688Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.004016641Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.006747406Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.009557173Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.012145745Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.01481984Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T22:58:10.068 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.017679674Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.020402082Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.02310695Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.025931191Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.028630027Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.031232528Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.03394095Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.03669477Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.03951325Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.04222641Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.044752946Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.047424453Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.049867684Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T22:58:10.069 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.052558141Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T22:58:10.106 INFO:teuthology.orchestra.run.smithi107.stdout:Backend: cephadm 2024-03-11T22:58:10.106 INFO:teuthology.orchestra.run.smithi107.stdout:Available: Yes 2024-03-11T22:58:10.107 INFO:teuthology.orchestra.run.smithi107.stdout:Paused: No 2024-03-11T22:58:10.304 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.055152952Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T22:58:10.304 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.057939458Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.060509576Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.063090526Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.06570116Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.068497025Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.071350845Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.074925828Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.078577275Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.081473407Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.083896458Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.08651709Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.087302392Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.088108592Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.089076573Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T22:58:10.305 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.089752651Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.090540016Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.091405763Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.09228219Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.093116843Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.0940999Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.094743422Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.095588857Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.096441432Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.097288703Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.09811963Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.098957066Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.09981046Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.100660863Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T22:58:10.306 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.101478029Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T22:58:10.307 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.102448167Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T22:58:10.307 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.103178963Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T22:58:10.307 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.104064879Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T22:58:10.307 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.104947144Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T22:58:10.307 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.105807525Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.106719474Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.107598997Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.108476558Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.109318227Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.110626088Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.111479187Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.112827512Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.113666339Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T22:58:10.308 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.115304706Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.116148958Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.11701062Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.118394566Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.121120068Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.123805209Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.126543992Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.129265669Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.132002667Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.134727746Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.137457505Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.137576049Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.140091806Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.142945726Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.145632269Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T22:58:10.309 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.1458528Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.146105383Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.148366747Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.151052626Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.153908358Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.156638243Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.159371452Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.162110711Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.164988047Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.167654577Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.170266954Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.172952629Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.180979034Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T22:58:10.310 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.183401663Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.186051546Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.194117326Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.201158066Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.201993204Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.202907564Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.203451582Z level=info msg="Executing migration" id="create permission table" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.204040404Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.204918747Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.205743552Z level=info msg="Executing migration" id="create role table" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.20658785Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.208018604Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.209514847Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.210405065Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.211254218Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T22:58:10.311 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.212102634Z level=info msg="Executing migration" id="create team role table" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.215024802Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.217632072Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.220308212Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.223172695Z level=info msg="Executing migration" id="create user role table" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.2258728Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.228459439Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.231097755Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.233747602Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.236717742Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.239539118Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.242997799Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.245824919Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.248462157Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.251355421Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T22:58:10.312 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.25189036Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.252362612Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.25276921Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.253238234Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.254346812Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.257204894Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.25780058Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.258188835Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.258589729Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.259050072Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.259546647Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.259887813Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.260306563Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.260743427Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T22:58:10.313 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.261226012Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.262255477Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.26258479Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.263027823Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.263491991Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.263945731Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.264940255Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.265525307Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.266071882Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.266183049Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.266485075Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.266585191Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.268307104Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.268775477Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T22:58:10.314 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.271042477Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.273676039Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.276209546Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.278779279Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.281390889Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.284137411Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.286689783Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.289275441Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.291864776Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.294411157Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.29766933Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.300362325Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T22:58:10.315 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.303088342Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T22:58:10.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:09 smithi195 ceph-mon[27640]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:10.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:09 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:09 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:09 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:09 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:09 smithi107 ceph-mon[25849]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:10.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:09 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:09 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:09 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:09 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.305645674Z level=info msg="Executing migration" id="add share column" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.30814503Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.31091636Z level=info msg="Executing migration" id="create file table" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.313628388Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.316173717Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.316913481Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.317383321Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.318181007Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T22:58:10.561 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.318964657Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.319776522Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.320542377Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.321364018Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.322518375Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.323365105Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.324166088Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.324938757Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.325780253Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.326599449Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.328201834Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.329539159Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T22:58:10.562 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.330364625Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.341990829Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.342698746Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.343532077Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.347311364Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.348109377Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.349062513Z level=info msg="Executing migration" id="create folder table" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.349821926Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.350679318Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.351525149Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.352401905Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=migrator t=2024-03-11T22:58:10.353232534Z level=info msg="migrations completed" performed=485 skipped=0 duration=720.982101ms 2024-03-11T22:58:10.563 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=sqlstore t=2024-03-11T22:58:10.353679771Z level=info msg="Created default organization" 2024-03-11T22:58:10.564 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=plugin.loader t=2024-03-11T22:58:10.420082392Z level=info msg="Plugin registered" pluginID=input 2024-03-11T22:58:10.564 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=plugin.loader t=2024-03-11T22:58:10.453694164Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T22:58:10.564 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=plugin.loader t=2024-03-11T22:58:10.453736134Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T22:58:10.564 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=secrets t=2024-03-11T22:58:10.453778537Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T22:58:10.564 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=query_data t=2024-03-11T22:58:10.456068787Z level=info msg="Query Service initialization" 2024-03-11T22:58:10.564 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=live.push_http t=2024-03-11T22:58:10.458779154Z level=info msg="Live Push Gateway initialization" 2024-03-11T22:58:10.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-11T22:58:11.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-mon[27544]: Deploying daemon node-exporter.a on smithi107 2024-03-11T22:58:11.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:10 smithi131 ceph-mon[27544]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:11.067 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:11.262 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:10 smithi107 ceph-mon[25849]: Deploying daemon node-exporter.a on smithi107 2024-03-11T22:58:11.262 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:10 smithi107 ceph-mon[25849]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:11.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:10 smithi195 ceph-mon[27640]: Deploying daemon node-exporter.a on smithi107 2024-03-11T22:58:11.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:10 smithi195 ceph-mon[27640]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:11.545 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=infra.usagestats.collector t=2024-03-11T22:58:11.150112177Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T22:58:11.545 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=provisioning.datasources t=2024-03-11T22:58:11.151119444Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T22:58:11.545 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=provisioning.datasources t=2024-03-11T22:58:11.162194795Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T22:58:11.545 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=provisioning.alerting t=2024-03-11T22:58:11.172965039Z level=info msg="starting to provision alerting" 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=provisioning.alerting t=2024-03-11T22:58:11.173001463Z level=info msg="finished to provision alerting" 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=ngalert.state.manager t=2024-03-11T22:58:11.173081563Z level=info msg="Warming state cache for startup" 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=grafanaStorageLogger t=2024-03-11T22:58:11.173097207Z level=info msg="storage starting" 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=ngalert.state.manager t=2024-03-11T22:58:11.17327063Z level=info msg="State cache has been initialized" states=0 duration=190.146µs 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=ticker t=2024-03-11T22:58:11.173340564Z level=info msg=starting first_tick=2024-03-11T22:58:20Z 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=ngalert.multiorg.alertmanager t=2024-03-11T22:58:11.173359563Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=http.server t=2024-03-11T22:58:11.174930616Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=sqlstore.transactions t=2024-03-11T22:58:11.183614162Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T22:58:11.546 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a[38814]: logger=sqlstore.transactions t=2024-03-11T22:58:11.195151108Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-11T22:58:12.132 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:11 smithi107 ceph-mon[25849]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:12.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:11 smithi131 ceph-mon[27544]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:12.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:11 smithi195 ceph-mon[27640]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:13.646 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:13 smithi131 ceph-mon[27544]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:13.646 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:13 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:13.701 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:13 smithi107 ceph-mon[25849]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:13.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:13 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:13.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:13 smithi195 ceph-mon[27640]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:13.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:13 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:14.701 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:58:14 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-a[51005]: ts=2024-03-11T22:58:14.617Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:58:14.976 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:58:14 smithi107 bash[50774]: 8bbd910ee2f1178d7873282f7c393dbe21952f7edf1c0ed5d5f5b16032ac706a 2024-03-11T22:58:14.977 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:58:14 smithi107 systemd[1]: Started Ceph node-exporter.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:15.623 INFO:teuthology.orchestra.run.smithi107.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-11T22:58:15.625 INFO:teuthology.orchestra.run.smithi107.stdout:agent.smithi107 smithi107 running 12s ago 3m - - 2024-03-11T22:58:15.625 INFO:teuthology.orchestra.run.smithi107.stdout:agent.smithi131 smithi131 running 0s ago 3m - - 2024-03-11T22:58:15.625 INFO:teuthology.orchestra.run.smithi107.stdout:agent.smithi195 smithi195 running 14s ago 3m - - 2024-03-11T22:58:15.625 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager.a smithi107 *:9093,9094 running (22s) 12s ago 21s 17.8M - 0.25.0 c8568f914cd2 9a55d7369b8e 2024-03-11T22:58:15.625 INFO:teuthology.orchestra.run.smithi107.stdout:grafana.a smithi131 *:3000 running (6s) 0s ago 6s 38.9M - 9.4.12 a0ea5e3c46d9 c5739fa70970 2024-03-11T22:58:15.625 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.x smithi131 *:8443,8765 running (2m) 0s ago 2m 449M - 19.0.0-1607-g94a53970 a6477a474d71 2df2ccf01faa 2024-03-11T22:58:15.625 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.y smithi107 *:9283,8765 running (4m) 12s ago 4m 500M - 19.0.0-1607-g94a53970 a6477a474d71 234ee877afdf 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:mgr.z smithi195 *:8443,8765 running (2m) 14s ago 2m 448M - 19.0.0-1607-g94a53970 a6477a474d71 7043afe03d05 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:mon.a smithi107 running (4m) 12s ago 4m 49.1M 2048M 19.0.0-1607-g94a53970 a6477a474d71 942d73530b9a 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:mon.b smithi131 running (3m) 0s ago 3m 37.0M 2048M 19.0.0-1607-g94a53970 a6477a474d71 6b411d0d6e6d 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:mon.c smithi195 running (3m) 14s ago 3m 37.4M 2048M 19.0.0-1607-g94a53970 a6477a474d71 ac7e9e7255a6 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter.a smithi107 *:9100 starting - - - - 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 smithi107 running (2m) 12s ago 2m 62.1M 4096M 19.0.0-1607-g94a53970 a6477a474d71 f1e563920eec 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 smithi131 running (2m) 0s ago 2m 59.5M 4096M 19.0.0-1607-g94a53970 a6477a474d71 cfa57fb12890 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 smithi195 running (92s) 14s ago 92s 59.5M 4096M 19.0.0-1607-g94a53970 a6477a474d71 23bc1d832960 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus.a smithi131 *:9095 running (51s) 0s ago 51s 28.5M - 2.43.0 a07b618ecd1d 4116e8e9bc5b 2024-03-11T22:58:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:rgw.foo.a smithi107 *:80 running (74s) 12s ago 73s 76.2M - 19.0.0-1607-g94a53970 a6477a474d71 998ec625860a 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: Deploying daemon node-exporter.b on smithi131 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.848 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:15 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:15.951 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: Deploying daemon node-exporter.b on smithi131 2024-03-11T22:58:15.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:15.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:15 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:16.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: Deploying daemon node-exporter.b on smithi131 2024-03-11T22:58:16.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.079 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:15 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:16.215 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:16 smithi131 systemd[1]: Starting Ceph node-exporter.b for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:16.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-11T22:58:16.559 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:16 smithi131 bash[39828]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T22:58:17.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:16 smithi131 ceph-mon[27544]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:17.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:16 smithi131 bash[39828]: Getting image source signatures 2024-03-11T22:58:17.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:16 smithi131 bash[39828]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T22:58:17.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:16 smithi131 bash[39828]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T22:58:17.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:16 smithi131 bash[39828]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T22:58:17.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:16 smithi195 ceph-mon[27640]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:17.092 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:16 smithi107 ceph-mon[25849]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:17.176 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:18.056 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:17 smithi131 ceph-mon[27544]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:18.056 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:17 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:18.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:17 smithi195 ceph-mon[27640]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:18.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:17 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:18.121 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:17 smithi107 ceph-mon[25849]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:18.122 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:17 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:18.309 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 bash[39828]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T22:58:18.309 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 bash[39828]: Writing manifest to image destination 2024-03-11T22:58:18.804 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 podman[39828]: 2024-03-11 22:58:18.475575056 +0000 UTC m=+2.358014574 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:58:18.804 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 podman[39828]: 2024-03-11 22:58:18.582584847 +0000 UTC m=+2.465024398 container create 4fa644216e24f35033fc0e27f08579ecdff4e8792166a1269e4bcb836d78942b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:58:18.924 INFO:teuthology.orchestra.run.smithi107.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-11T22:58:18.924 INFO:teuthology.orchestra.run.smithi107.stdout:agent 3/3 17s ago 46s * 2024-03-11T22:58:18.924 INFO:teuthology.orchestra.run.smithi107.stdout:alertmanager ?:9093,9094 1/1 16s ago 71s smithi107=a;count:1 2024-03-11T22:58:18.925 INFO:teuthology.orchestra.run.smithi107.stdout:grafana ?:3000 1/1 3s ago 69s smithi131=a;count:1 2024-03-11T22:58:18.925 INFO:teuthology.orchestra.run.smithi107.stdout:mgr 3/3 17s ago 2m smithi107=y;smithi131=x;smithi195=z;count:3 2024-03-11T22:58:18.925 INFO:teuthology.orchestra.run.smithi107.stdout:mon 3/3 17s ago 3m smithi107:172.21.15.107=a;smithi131:172.21.15.131=b;smithi195:172.21.15.195=c;count:3 2024-03-11T22:58:18.925 INFO:teuthology.orchestra.run.smithi107.stdout:node-exporter ?:9100 0/3 - 74s smithi107=a;smithi131=b;smithi195=c;count:3 2024-03-11T22:58:18.925 INFO:teuthology.orchestra.run.smithi107.stdout:osd 3 17s ago - 2024-03-11T22:58:18.925 INFO:teuthology.orchestra.run.smithi107.stdout:prometheus ?:9095 1/1 3s ago 77s smithi131=a;count:1 2024-03-11T22:58:18.925 INFO:teuthology.orchestra.run.smithi107.stdout:rgw.foo.a ?:80 1/1 16s ago 76s smithi107=foo.a;count:1 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 podman[39828]: 2024-03-11 22:58:18.80115207 +0000 UTC m=+2.683591576 container init 4fa644216e24f35033fc0e27f08579ecdff4e8792166a1269e4bcb836d78942b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 podman[39828]: 2024-03-11 22:58:18.806567576 +0000 UTC m=+2.689007081 container start 4fa644216e24f35033fc0e27f08579ecdff4e8792166a1269e4bcb836d78942b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.810Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.810Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.810Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.810Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:58:19.060 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:58:19.061 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:58:19.062 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:58:19.063 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.811Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.812Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b[39880]: ts=2024-03-11T22:58:18.812Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 bash[39828]: 4fa644216e24f35033fc0e27f08579ecdff4e8792166a1269e4bcb836d78942b 2024-03-11T22:58:19.064 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:58:18 smithi131 systemd[1]: Started Ceph node-exporter.b for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:19.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-11T22:58:19.860 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:20.110 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:19 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ::ffff:172.21.15.131 - - [11/Mar/2024:22:58:19] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:58:20.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:19 smithi107 ceph-mon[25849]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:20.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:19 smithi107 ceph-mon[25849]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:20.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:19 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:19 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:19 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.110 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:19 smithi107 ceph-mon[25849]: Deploying daemon node-exporter.c on smithi195 2024-03-11T22:58:20.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:19 smithi195 ceph-mon[27640]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:20.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:19 smithi195 ceph-mon[27640]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:20.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:19 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:19 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:19 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.215 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:19 smithi195 ceph-mon[27640]: Deploying daemon node-exporter.c on smithi195 2024-03-11T22:58:20.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:19 smithi131 ceph-mon[27544]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:20.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:19 smithi131 ceph-mon[27544]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:20.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:19 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:19 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:19 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:20.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:19 smithi131 ceph-mon[27544]: Deploying daemon node-exporter.c on smithi195 2024-03-11T22:58:22.050 INFO:teuthology.orchestra.run.smithi107.stdout:HOST ADDR LABELS STATUS 2024-03-11T22:58:22.051 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 172.21.15.107 2024-03-11T22:58:22.051 INFO:teuthology.orchestra.run.smithi107.stdout:smithi131 172.21.15.131 2024-03-11T22:58:22.051 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 172.21.15.195 2024-03-11T22:58:22.051 INFO:teuthology.orchestra.run.smithi107.stdout:3 hosts in cluster 2024-03-11T22:58:22.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:22 smithi107 ceph-mon[25849]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:22.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:22 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:22 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.445 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:22 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.559 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:22 smithi131 ceph-mon[27544]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:22 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:22 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:22 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:22 smithi195 ceph-mon[27640]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:22 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:22 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:22 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:22.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-11T22:58:23.042 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:22 smithi195 bash[39700]: Writing manifest to image destination 2024-03-11T22:58:23.043 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 podman[39700]: 2024-03-11 22:58:23.005982197 +0000 UTC m=+2.391671571 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T22:58:23.151 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:23.292 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 podman[39700]: 2024-03-11 22:58:23.103753578 +0000 UTC m=+2.489442952 container create 1f0bf101b8efc561f701dfbc756e59577253ad98df2363e4939544a58611352f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:58:23.293 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-mon[27640]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:23.293 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.293 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.293 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.407 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:23 smithi107 ceph-mon[25849]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:23.408 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:23 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.408 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:23 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.408 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:23 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.560 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:23 smithi131 ceph-mon[27544]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:23.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:23 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:23 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.561 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:23 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:23.579 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 podman[39700]: 2024-03-11 22:58:23.289127923 +0000 UTC m=+2.674817312 container init 1f0bf101b8efc561f701dfbc756e59577253ad98df2363e4939544a58611352f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:58:23.579 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 podman[39700]: 2024-03-11 22:58:23.294639096 +0000 UTC m=+2.680328473 container start 1f0bf101b8efc561f701dfbc756e59577253ad98df2363e4939544a58611352f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:58:23.580 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.297Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T22:58:23.580 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.297Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T22:58:23.580 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.298Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T22:58:23.580 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.298Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T22:58:23.580 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.298Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T22:58:23.580 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.298Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T22:58:23.580 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.298Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T22:58:23.581 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.298Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T22:58:23.581 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T22:58:23.581 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T22:58:23.581 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T22:58:23.581 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T22:58:23.582 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T22:58:23.583 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T22:58:23.584 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T22:58:23.585 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c[40021]: ts=2024-03-11T22:58:23.299Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 bash[39700]: 1f0bf101b8efc561f701dfbc756e59577253ad98df2363e4939544a58611352f 2024-03-11T22:58:23.586 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:58:23 smithi195 systemd[1]: Started Ceph node-exporter.c for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.267 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.268 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.268 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.268 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:24.268 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:24.268 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:24.268 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:24 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:24.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:24.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:24.329 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:24 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:24.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:24.387 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T22:58:24.388 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:24 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:25.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:25 smithi131 ceph-mon[27544]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:25.393 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:25 smithi131 ceph-mon[27544]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:58:25.394 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:25 smithi131 ceph-mon[27544]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T22:58:25.394 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:25 smithi131 ceph-mon[27544]: Reconfiguring daemon alertmanager.a on smithi107 2024-03-11T22:58:25.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:25 smithi195 ceph-mon[27640]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:25.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:25 smithi195 ceph-mon[27640]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:58:25.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:25 smithi195 ceph-mon[27640]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T22:58:25.433 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:25 smithi195 ceph-mon[27640]: Reconfiguring daemon alertmanager.a on smithi107 2024-03-11T22:58:25.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:25 smithi107 ceph-mon[25849]: pgmap v30: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:25.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:25 smithi107 ceph-mon[25849]: Metadata not up to date on all hosts. Skipping non agent specs 2024-03-11T22:58:25.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:25 smithi107 ceph-mon[25849]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T22:58:25.451 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:25 smithi107 ceph-mon[25849]: Reconfiguring daemon alertmanager.a on smithi107 2024-03-11T22:58:27.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:27 smithi107 ceph-mon[25849]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:27.430 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:27 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:27.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:27 smithi131 ceph-mon[27544]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:27.446 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:27 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:27.468 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:27 smithi195 ceph-mon[27640]: pgmap v31: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:27.468 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:27 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:28.637 INFO:teuthology.orchestra.run.smithi107.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400NK400BGN 372G No 5s ago Has a FileSystem, LVM detected 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme1n1 ssd Linux_56673ee7cf6493dbf5f3 89.4G Yes 5s ago 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme2n1 ssd Linux_4ee21345192a9b55fa81 89.4G Yes 5s ago 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme3n1 ssd Linux_6ad205e36222af176688 89.4G Yes 5s ago 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/nvme4n1 ssd Linux_1e0d947f7373791b0782 89.4G No 5s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi107 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5R0J2 931G No 5s ago Has partitions 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi131 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300N4400BGN 372G No 1s ago Has a FileSystem, LVM detected 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi131 /dev/nvme1n1 ssd Linux_0248146d58258476bc83 89.4G Yes 1s ago 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi131 /dev/nvme2n1 ssd Linux_f4df03a568b227a4fd1a 89.4G Yes 1s ago 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi131 /dev/nvme3n1 ssd Linux_91a01e0503724f1d4f5c 89.4G Yes 1s ago 2024-03-11T22:58:28.638 INFO:teuthology.orchestra.run.smithi107.stdout:smithi131 /dev/nvme4n1 ssd Linux_ccae4a2987ac0f5f25d0 89.4G No 1s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:58:28.639 INFO:teuthology.orchestra.run.smithi107.stdout:smithi131 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XK3P 931G No 1s ago Has partitions 2024-03-11T22:58:28.639 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB751300FS480DGN 447G No 4s ago Has a FileSystem, LVM detected 2024-03-11T22:58:28.639 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 /dev/nvme1n1 ssd Linux_f7dc753a0085e673bb00 89.4G Yes 4s ago 2024-03-11T22:58:28.639 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 /dev/nvme2n1 ssd Linux_6e1b1dd72cad41d41a25 89.4G Yes 4s ago 2024-03-11T22:58:28.639 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 /dev/nvme3n1 ssd Linux_3a4215828f25f37a686c 89.4G Yes 4s ago 2024-03-11T22:58:28.639 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 /dev/nvme4n1 ssd Linux_a2c17452542ea6daf680 89.4G No 4s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-03-11T22:58:28.639 INFO:teuthology.orchestra.run.smithi107.stdout:smithi195 /dev/sda hdd ATA_ST1000NM0033-9ZM173_Z1W5XLZZ 931G No 4s ago Has partitions 2024-03-11T22:58:28.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:28 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:28.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:28 smithi131 ceph-mon[27544]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:28.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:28 smithi131 ceph-mon[27544]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:28.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:28 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:28.841 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:28 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:28.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:28 smithi107 ceph-mon[25849]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:28.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:28 smithi107 ceph-mon[25849]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:28.842 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:28 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:28.869 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:28 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:28.869 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:28 smithi195 ceph-mon[27640]: pgmap v32: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:28.869 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:28 smithi195 ceph-mon[27640]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:28.869 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:28 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:29.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-03-11T22:58:30.202 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:29 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ::ffff:172.21.15.131 - - [11/Mar/2024:22:58:29] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:58:30.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:29 smithi107 systemd[1]: Stopping Ceph alertmanager.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:30.251 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:30.649 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[48882]: ts=2024-03-11T22:58:30.407Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:58:30.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.650 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:30 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:30 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:30 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:30 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:30 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:30 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:30 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:30 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:30 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:30 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:30 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:30 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:30 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:30 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:30 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:30.910 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:30 smithi107 podman[52744]: 2024-03-11 22:58:30.656653588 +0000 UTC m=+0.389732514 container died 9a55d7369b8e2dcb38b21eb36c9932b695f374ff38a0b46d42ca96f4ce3a22c3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:31.201 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:31 smithi107 podman[52744]: 2024-03-11 22:58:31.089233309 +0000 UTC m=+0.822312388 container cleanup 9a55d7369b8e2dcb38b21eb36c9932b695f374ff38a0b46d42ca96f4ce3a22c3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:31.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:31 smithi107 bash[52744]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T22:58:31.559 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:31 smithi131 ceph-mon[27544]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:31.702 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:31 smithi107 ceph-mon[25849]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:31.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:31 smithi195 ceph-mon[27640]: pgmap v33: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:32.525 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:32 smithi107 ceph-mon[25849]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:32.525 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:32 smithi107 podman[52792]: 2024-03-11 22:58:32.2702093 +0000 UTC m=+1.840320846 container remove 9a55d7369b8e2dcb38b21eb36c9932b695f374ff38a0b46d42ca96f4ce3a22c3 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:32.525 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:32 smithi107 podman[52792]: 2024-03-11 22:58:32.370794383 +0000 UTC m=+1.940905928 volume remove fea38db705cc909cee5221dc837cc98a1955057eb9835b7fd8d61c1dc768f3da 2024-03-11T22:58:32.525 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:32 smithi107 bash[52974]: Error: no container with name or ID "ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a" found: no such container 2024-03-11T22:58:32.525 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:32 smithi107 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T22:58:32.525 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:32 smithi107 systemd[1]: Stopped Ceph alertmanager.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:32.525 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:32 smithi107 systemd[1]: Starting Ceph alertmanager.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:32.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:32 smithi131 ceph-mon[27544]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:32.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:32 smithi195 ceph-mon[27640]: pgmap v34: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:33.117 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: agent 2024-03-11T22:58:33.117 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: agent 2024-03-11T22:58:33.117 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.117 INFO:teuthology.orchestra.run.smithi107.stdout: host_pattern: '*' 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:57:32.531460Z' 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:22.971124Z' 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout: running: 3 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout: size: 3 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout:events: 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout:- 2024-03-11T22:57:33.944587Z service:agent [INFO] "service was created" 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: alertmanager 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: alertmanager 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout: count: 1 2024-03-11T22:58:33.118 INFO:teuthology.orchestra.run.smithi107.stdout: hosts: 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi107=a 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:57:07.026655Z' 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:22.971346Z' 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: ports: 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: - 9093 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: - 9094 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: running: 1 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout: size: 1 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:events: 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:- 2024-03-11T22:57:54.664004Z service:alertmanager [INFO] "service was created" 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: grafana 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: grafana 2024-03-11T22:58:33.119 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: count: 1 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: hosts: 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi131=a 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout:spec: 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: anonymous_access: true 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: protocol: https 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:57:09.657564Z' 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:29.618382Z' 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: ports: 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: - 3000 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: running: 1 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout: size: 1 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout:events: 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout:- 2024-03-11T22:58:09.621034Z service:grafana [INFO] "service was created" 2024-03-11T22:58:33.120 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: mgr 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: mgr 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: count: 3 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: hosts: 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi107=y 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi131=x 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi195=z 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:55:25.756503Z' 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:22.971068Z' 2024-03-11T22:58:33.121 INFO:teuthology.orchestra.run.smithi107.stdout: running: 3 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: size: 3 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: mon 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: mon 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: count: 3 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: hosts: 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi107:172.21.15.107=a 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi131:172.21.15.131=b 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi195:172.21.15.195=c 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:54:58.899815Z' 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:22.970958Z' 2024-03-11T22:58:33.122 INFO:teuthology.orchestra.run.smithi107.stdout: running: 3 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: size: 3 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: node-exporter 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: node-exporter 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: count: 3 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: hosts: 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi107=a 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi131=b 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi195=c 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:57:04.315813Z' 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:29.346741Z' 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: ports: 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: - 9100 2024-03-11T22:58:33.123 INFO:teuthology.orchestra.run.smithi107.stdout: running: 2 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout: size: 3 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:events: 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:- 2024-03-11T22:58:23.464333Z service:node-exporter [INFO] "service was created" 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: osd 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: osd 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:unmanaged: true 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:spec: 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout: filter_logic: AND 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout: objectstore: bluestore 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout: container_image_id: a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:22.971200Z' 2024-03-11T22:58:33.124 INFO:teuthology.orchestra.run.smithi107.stdout: running: 3 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: size: 3 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: prometheus 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: prometheus 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: count: 1 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: hosts: 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi131=a 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:57:01.715975Z' 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:29.618313Z' 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: ports: 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: - 9095 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: running: 1 2024-03-11T22:58:33.125 INFO:teuthology.orchestra.run.smithi107.stdout: size: 1 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:events: 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:- 2024-03-11T22:58:23.472140Z service:prometheus [INFO] "service was created" 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:--- 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:service_type: rgw 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:service_id: foo.a 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:service_name: rgw.foo.a 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:placement: 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout: count: 1 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout: hosts: 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout: - smithi107=foo.a 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout:status: 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout: created: '2024-03-11T22:57:02.008531Z' 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout: last_refresh: '2024-03-11T22:58:22.971273Z' 2024-03-11T22:58:33.126 INFO:teuthology.orchestra.run.smithi107.stdout: ports: 2024-03-11T22:58:33.127 INFO:teuthology.orchestra.run.smithi107.stdout: - 80 2024-03-11T22:58:33.127 INFO:teuthology.orchestra.run.smithi107.stdout: running: 1 2024-03-11T22:58:33.127 INFO:teuthology.orchestra.run.smithi107.stdout: size: 1 2024-03-11T22:58:33.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:33 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:33.686 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:33 smithi107 ceph-mon[25849]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:58:33.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:33 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:33.810 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:33 smithi131 ceph-mon[27544]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:58:33.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:33 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T22:58:33.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:33 smithi195 ceph-mon[27640]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-03-11T22:58:34.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-11T22:58:34.377 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 podman[53194]: 2024-03-11 22:58:34.256981154 +0000 UTC m=+0.539147700 volume create 53febce83a6e223c06d87ded58ce16e964293a66a83bec398e312664dd2943b1 2024-03-11T22:58:34.378 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 podman[53194]: 2024-03-11 22:58:34.166665933 +0000 UTC m=+0.448832486 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T22:58:34.378 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 podman[53194]: 2024-03-11 22:58:34.37418783 +0000 UTC m=+0.656354410 container create 88d11f1026ecbd1c8cb4c66b8ecd806b3a3ff524292839e137407aecb38e7348 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:34.688 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-mon[25849]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:34.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:34 smithi131 ceph-mon[27544]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:34.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:34 smithi195 ceph-mon[27640]: pgmap v35: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:34.834 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 podman[53194]: 2024-03-11 22:58:34.685223393 +0000 UTC m=+0.967389951 container init 88d11f1026ecbd1c8cb4c66b8ecd806b3a3ff524292839e137407aecb38e7348 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 podman[53194]: 2024-03-11 22:58:34.690313844 +0000 UTC m=+0.972480383 container start 88d11f1026ecbd1c8cb4c66b8ecd806b3a3ff524292839e137407aecb38e7348 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.702Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.702Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.702Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.107 port=9094 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.703Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.727Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.727Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.729Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:34.729Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 bash[53194]: 88d11f1026ecbd1c8cb4c66b8ecd806b3a3ff524292839e137407aecb38e7348 2024-03-11T22:58:34.952 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:34 smithi107 systemd[1]: Started Ceph alertmanager.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:36.059 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:35 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:36.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:35 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:36.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:35 smithi131 ceph-mon[27544]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T22:58:36.060 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:35 smithi131 ceph-mon[27544]: Reconfiguring daemon prometheus.a on smithi131 2024-03-11T22:58:36.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:35 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:36.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:35 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:36.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:35 smithi107 ceph-mon[25849]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T22:58:36.202 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:35 smithi107 ceph-mon[25849]: Reconfiguring daemon prometheus.a on smithi131 2024-03-11T22:58:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:35 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:35 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:35 smithi195 ceph-mon[27640]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T22:58:36.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:35 smithi195 ceph-mon[27640]: Reconfiguring daemon prometheus.a on smithi131 2024-03-11T22:58:36.985 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:36 smithi107 ceph-mon[25849]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:36.986 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:36 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:36.704Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000818923s 2024-03-11T22:58:37.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:36 smithi131 ceph-mon[27544]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:37.328 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:36 smithi195 ceph-mon[27640]: pgmap v36: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:37.890 INFO:teuthology.orchestra.run.smithi107.stdout:osd 3 14s ago - 2024-03-11T22:58:38.201 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:38 smithi107 ceph-mon[25849]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:38.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:38 smithi131 ceph-mon[27544]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:38.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:38 smithi195 ceph-mon[27640]: from='mgr.14430 172.21.15.107:0/3402025515' entity='mgr.y' 2024-03-11T22:58:38.782 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T22:58:38.792 INFO:tasks.cephadm:Teardown begin 2024-03-11T22:58:38.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:58:38.827 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:58:38.863 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:58:38.896 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T22:58:38.897 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:58:38.911 DEBUG:teuthology.orchestra.run.smithi131:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:58:38.928 DEBUG:teuthology.orchestra.run.smithi195:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T22:58:38.952 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T22:58:38.952 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T22:58:38.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a 2024-03-11T22:58:39.030 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:38 smithi107 systemd[1]: Stopping Ceph mon.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:39.290 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-mon[25849]: pgmap v37: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:39.291 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-mon[25849]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:39.450 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:39 smithi131 ceph-mon[27544]: pgmap v37: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:39.450 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:39 smithi131 ceph-mon[27544]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:39.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:39 smithi195 ceph-mon[27640]: pgmap v37: 129 pgs: 129 active+clean; 582 KiB data, 98 MiB used, 268 GiB / 268 GiB avail 2024-03-11T22:58:39.578 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:39 smithi195 ceph-mon[27640]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a[25826]: 2024-03-11T22:58:39.676+0000 7f0ca6e51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a[25826]: 2024-03-11T22:58:39.676+0000 7f0ca6e51700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:39 smithi107 podman[53904]: 2024-03-11 22:58:39.697989428 +0000 UTC m=+0.483913414 container died 942d73530b9a87182bae861e4d2296177a95705c40e63f48139c1fd732e57460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, ceph=True) 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:39] ENGINE Bus STOPPING 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:39] ENGINE Bus STOPPED 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:39] ENGINE Bus STARTING 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:39] ENGINE Serving on http://:::9283 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:39] ENGINE Bus STARTED 2024-03-11T22:58:39.952 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:39 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: ::ffff:172.21.15.131 - - [11/Mar/2024:22:58:39] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T22:58:40.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:40 smithi107 podman[53904]: 2024-03-11 22:58:40.191892413 +0000 UTC m=+0.977816443 container cleanup 942d73530b9a87182bae861e4d2296177a95705c40e63f48139c1fd732e57460 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-11T22:58:40.421 INFO:journalctl@ceph.mon.a.smithi107.stdout:Mar 11 22:58:40 smithi107 bash[53904]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-a 2024-03-11T22:58:40.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T22:58:40.589 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:58:40.589 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T22:58:40.589 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-11T22:58:40.589 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.b 2024-03-11T22:58:40.676 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:40 smithi131 systemd[1]: Stopping Ceph mon.b for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:43.050 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:42 smithi131 systemd[1]: Stopping Ceph prometheus.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:43.309 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b[27521]: 2024-03-11T22:58:43.045+0000 7f718faca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:58:43.310 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b[27521]: 2024-03-11T22:58:43.045+0000 7f718faca700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.162Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.162Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.162Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.162Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.162Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.162Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.163Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T22:58:43.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.163Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T22:58:43.311 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.163Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T22:58:43.311 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.163Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T22:58:43.311 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.163Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T22:58:43.311 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[36109]: ts=2024-03-11T22:58:43.163Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T22:58:43.311 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 podman[41101]: 2024-03-11 22:58:43.173547421 +0000 UTC m=+0.099856060 container died 4116e8e9bc5bfb5015415796c121d0ad19cd9bcdacf273a386d1ee8541536da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:43.451 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:43] ENGINE Bus STOPPING 2024-03-11T22:58:43.451 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T22:58:43.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:43] ENGINE Bus STOPPED 2024-03-11T22:58:43.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:43] ENGINE Bus STARTING 2024-03-11T22:58:43.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:43] ENGINE Serving on http://:::9283 2024-03-11T22:58:43.452 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:43 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: [11/Mar/2024:22:58:43] ENGINE Bus STARTED 2024-03-11T22:58:43.809 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:43 smithi131 podman[41071]: 2024-03-11 22:58:43.558870026 +0000 UTC m=+0.614702704 container died 6b411d0d6e6d2e89d9e1aed47ce930d775ecb45d4c5bc786df073cc816b13be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-11T22:58:43.810 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 podman[41101]: 2024-03-11 22:58:43.535047978 +0000 UTC m=+0.461356616 container cleanup 4116e8e9bc5bfb5015415796c121d0ad19cd9bcdacf273a386d1ee8541536da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:43.810 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:43 smithi131 bash[41101]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T22:58:44.251 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:43 smithi131 podman[41071]: 2024-03-11 22:58:43.878538127 +0000 UTC m=+0.934370813 container cleanup 6b411d0d6e6d2e89d9e1aed47ce930d775ecb45d4c5bc786df073cc816b13be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T22:58:44.251 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:43 smithi131 bash[41071]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b 2024-03-11T22:58:44.251 INFO:journalctl@ceph.mon.b.smithi131.stdout:Mar 11 22:58:44 smithi131 podman[41131]: 2024-03-11 22:58:44.038066804 +0000 UTC m=+0.802987500 container remove 6b411d0d6e6d2e89d9e1aed47ce930d775ecb45d4c5bc786df073cc816b13be0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T22:58:44.397 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T22:58:44.447 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:58:44.448 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-11T22:58:44.448 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-11T22:58:44.448 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.c 2024-03-11T22:58:44.559 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 podman[41120]: 2024-03-11 22:58:44.247925515 +0000 UTC m=+1.073922310 container remove 4116e8e9bc5bfb5015415796c121d0ad19cd9bcdacf273a386d1ee8541536da0 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:44.560 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T22:58:44.560 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 systemd[1]: Stopped Ceph prometheus.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:44.560 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.218s CPU time. 2024-03-11T22:58:44.560 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 systemd[1]: Starting Ceph prometheus.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:44.750 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:44 smithi195 systemd[1]: Stopping Ceph mon.c for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:44.879 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 podman[41283]: 2024-03-11 22:58:44.512284023 +0000 UTC m=+0.018880955 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T22:58:44.880 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 podman[41283]: 2024-03-11 22:58:44.640951204 +0000 UTC m=+0.147548163 container create 164f3903a7a4adabf76d4a357861bb27f04aadc980866b16dbee2033f0d40f9f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:44.880 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 podman[41283]: 2024-03-11 22:58:44.876347202 +0000 UTC m=+0.382944116 container init 164f3903a7a4adabf76d4a357861bb27f04aadc980866b16dbee2033f0d40f9f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:45.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:44 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-c[27605]: 2024-03-11T22:58:44.745+0000 7f59226c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:58:45.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:44 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-c[27605]: 2024-03-11T22:58:44.745+0000 7f59226c9700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T22:58:45.078 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:44 smithi195 podman[40852]: 2024-03-11 22:58:44.981150479 +0000 UTC m=+0.340181849 container died ac7e9e7255a6b5bd46b058e30b57d04ccd3fedc23ec9a7882c6028bbeccd5c01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-11T22:58:45.178 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 podman[41283]: 2024-03-11 22:58:44.881636066 +0000 UTC m=+0.388232989 container start 164f3903a7a4adabf76d4a357861bb27f04aadc980866b16dbee2033f0d40f9f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.904Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.904Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.904Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi131 (none))" 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.904Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.904Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.905Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.906Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.907Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.907Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.909Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T22:58:45.179 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.909Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=5.849µs 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.909Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.909Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.910Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.910Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.116µs wal_replay_duration=1.178647ms wbl_replay_duration=159ns total_replay_duration=1.224325ms 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.911Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.911Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.911Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.929Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.772538ms db_storage=672ns remote_storage=2.097µs web_handler=344ns query_engine=630ns scrape=443.8µs scrape_sd=81.814µs notify=15.59µs notify_sd=15.467µs rules=17.005102ms tracing=6.06µs 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.929Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:58:44.929Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 bash[41283]: 164f3903a7a4adabf76d4a357861bb27f04aadc980866b16dbee2033f0d40f9f 2024-03-11T22:58:45.180 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:58:44 smithi131 systemd[1]: Started Ceph prometheus.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:58:45.201 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:58:44 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:58:44.707Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003385655s 2024-03-11T22:58:45.514 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:45 smithi195 podman[40852]: 2024-03-11 22:58:45.326567346 +0000 UTC m=+0.685598715 container cleanup ac7e9e7255a6b5bd46b058e30b57d04ccd3fedc23ec9a7882c6028bbeccd5c01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-11T22:58:45.515 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:45 smithi195 bash[40852]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-c 2024-03-11T22:58:45.828 INFO:journalctl@ceph.mon.c.smithi195.stdout:Mar 11 22:58:45 smithi195 podman[40871]: 2024-03-11 22:58:45.510956806 +0000 UTC m=+0.520539976 container remove ac7e9e7255a6b5bd46b058e30b57d04ccd3fedc23ec9a7882c6028bbeccd5c01 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-94a5397, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T22:58:47.084 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T22:58:47.128 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:58:47.128 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-11T22:58:47.128 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T22:58:47.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.y 2024-03-11T22:58:47.426 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:47 smithi107 systemd[1]: Stopping Ceph mgr.y for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:47.702 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:47 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:58:47.421+0000 7fddf9d27700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:58:47.702 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:47 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y[26107]: 2024-03-11T22:58:47.421+0000 7fddf9d27700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-11T22:58:47.702 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:47 smithi107 podman[54085]: 2024-03-11 22:58:47.453692686 +0000 UTC m=+0.142446359 container died 234ee877afdfc8c791ac2f75d2d84fc7113e9abe4ed6108839aa3d1d1b8a61d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-03-11T22:58:48.200 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:47 smithi107 podman[54085]: 2024-03-11 22:58:47.911651768 +0000 UTC m=+0.600405502 container cleanup 234ee877afdfc8c791ac2f75d2d84fc7113e9abe4ed6108839aa3d1d1b8a61d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:94a539707733325322964c6a187c6a1b41be1daf, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y, RELEASE=squid-94a5397, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-11T22:58:48.200 INFO:journalctl@ceph.mgr.y.smithi107.stdout:Mar 11 22:58:47 smithi107 bash[54085]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-y 2024-03-11T22:58:48.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T22:58:48.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:58:48.369 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T22:58:48.369 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T22:58:48.369 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.x 2024-03-11T22:58:48.559 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:58:48 smithi131 systemd[1]: Stopping Ceph mgr.x for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:49.618 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:58:49 smithi131 podman[41897]: 2024-03-11 22:58:49.349362739 +0000 UTC m=+0.439568903 container died 2df2ccf01faa260d3c21645299b09774a5b2ba5927ee18becc1ded62a4246d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-11T22:58:49.927 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:58:49 smithi131 podman[41897]: 2024-03-11 22:58:49.688834813 +0000 UTC m=+0.779040974 container cleanup 2df2ccf01faa260d3c21645299b09774a5b2ba5927ee18becc1ded62a4246d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T22:58:49.927 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:58:49 smithi131 bash[41897]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x 2024-03-11T22:58:50.238 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:58:49 smithi131 podman[41927]: 2024-03-11 22:58:49.923575926 +0000 UTC m=+0.808117300 container remove 2df2ccf01faa260d3c21645299b09774a5b2ba5927ee18becc1ded62a4246d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-11T22:58:50.238 INFO:journalctl@ceph.mgr.x.smithi131.stdout:Mar 11 22:58:49 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:58:50.260 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T22:58:50.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:58:50.301 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T22:58:50.301 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-11T22:58:50.301 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.z 2024-03-11T22:58:50.578 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:58:50 smithi195 systemd[1]: Stopping Ceph mgr.z for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:50.912 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:58:50 smithi195 podman[41526]: 2024-03-11 22:58:50.608236396 +0000 UTC m=+0.126075948 container died 7043afe03d0569606d0d462dd7b859e57b0f85a27a6db63c62b259ff08130f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-11T22:58:51.328 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:58:50 smithi195 podman[41526]: 2024-03-11 22:58:50.990556501 +0000 UTC m=+0.508396053 container cleanup 7043afe03d0569606d0d462dd7b859e57b0f85a27a6db63c62b259ff08130f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_BRANCH=HEAD) 2024-03-11T22:58:51.328 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:58:50 smithi195 bash[41526]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z 2024-03-11T22:58:51.328 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:58:51 smithi195 podman[41545]: 2024-03-11 22:58:51.150199823 +0000 UTC m=+0.539844837 container remove 7043afe03d0569606d0d462dd7b859e57b0f85a27a6db63c62b259ff08130f88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:58:51.328 INFO:journalctl@ceph.mgr.z.smithi195.stdout:Mar 11 22:58:51 smithi195 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:58:51.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@mgr.z.service' 2024-03-11T22:58:51.396 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:58:51.397 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-11T22:58:51.397 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T22:58:51.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.0 2024-03-11T22:58:51.701 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:51 smithi107 systemd[1]: Stopping Ceph osd.0 for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:58:52.421 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:52 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0[38174]: 2024-03-11T22:58:52.042+0000 7fa296de3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:58:52.421 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:52 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0[38174]: 2024-03-11T22:58:52.042+0000 7fa296de3700 -1 osd.0 31 *** Got signal Terminated *** 2024-03-11T22:58:52.421 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:52 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0[38174]: 2024-03-11T22:58:52.042+0000 7fa296de3700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:58:57.702 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:57 smithi107 podman[54444]: 2024-03-11 22:58:57.439653005 +0000 UTC m=+5.654764349 container died f1e563920eec5bf7c3c70f39570ec373d629458efff3f1403ce8b14f719d59f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:58:58.050 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:57 smithi107 podman[54444]: 2024-03-11 22:58:57.887206598 +0000 UTC m=+6.102317940 container cleanup f1e563920eec5bf7c3c70f39570ec373d629458efff3f1403ce8b14f719d59f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-94a5397) 2024-03-11T22:58:58.050 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:57 smithi107 bash[54444]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0 2024-03-11T22:58:58.451 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:58 smithi107 podman[54894]: 2024-03-11 22:58:58.046764213 +0000 UTC m=+0.602866190 container remove f1e563920eec5bf7c3c70f39570ec373d629458efff3f1403ce8b14f719d59f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-11T22:58:58.711 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:58 smithi107 podman[54993]: 2024-03-11 22:58:58.355529203 +0000 UTC m=+0.018300201 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:58:58.711 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:58 smithi107 podman[54993]: 2024-03-11 22:58:58.464050765 +0000 UTC m=+0.126821802 container create aa642caea7354c8ff50da7377c9eadfa3d9cfa11dc95febb9bab4ee2cfc504ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:58:58.712 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:58 smithi107 podman[54993]: 2024-03-11 22:58:58.70809259 +0000 UTC m=+0.370863585 container init aa642caea7354c8ff50da7377c9eadfa3d9cfa11dc95febb9bab4ee2cfc504ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-03-11T22:58:59.002 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:58 smithi107 podman[54993]: 2024-03-11 22:58:58.713349511 +0000 UTC m=+0.376120508 container start aa642caea7354c8ff50da7377c9eadfa3d9cfa11dc95febb9bab4ee2cfc504ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-11T22:58:59.002 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:58 smithi107 podman[54993]: 2024-03-11 22:58:58.782874739 +0000 UTC m=+0.445645736 container attach aa642caea7354c8ff50da7377c9eadfa3d9cfa11dc95febb9bab4ee2cfc504ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-11T22:58:59.297 INFO:journalctl@ceph.osd.0.smithi107.stdout:Mar 11 22:58:58 smithi107 podman[54993]: 2024-03-11 22:58:58.99912534 +0000 UTC m=+0.661896340 container died aa642caea7354c8ff50da7377c9eadfa3d9cfa11dc95febb9bab4ee2cfc504ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T22:58:59.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T22:58:59.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:58:59.631 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T22:58:59.631 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T22:58:59.631 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.1 2024-03-11T22:58:59.944 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:58:59 smithi131 systemd[1]: Stopping Ceph osd.1 for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:00.310 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:58:59 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1[33175]: 2024-03-11T22:58:59.939+0000 7f935de8f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:59:00.310 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:58:59 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1[33175]: 2024-03-11T22:58:59.939+0000 7f935de8f700 -1 osd.1 31 *** Got signal Terminated *** 2024-03-11T22:59:00.310 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:58:59 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1[33175]: 2024-03-11T22:58:59.939+0000 7f935de8f700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:59:05.672 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:05 smithi131 podman[42172]: 2024-03-11 22:59:05.370860416 +0000 UTC m=+5.546713825 container died cfa57fb1289017d96a4a6115f4a5649dab1314f6605d07b5626b14096e94157f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T22:59:05.997 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:05 smithi131 podman[42172]: 2024-03-11 22:59:05.750924417 +0000 UTC m=+5.926777823 container cleanup cfa57fb1289017d96a4a6115f4a5649dab1314f6605d07b5626b14096e94157f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T22:59:05.997 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:05 smithi131 bash[42172]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1 2024-03-11T22:59:06.306 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:05 smithi131 podman[42384]: 2024-03-11 22:59:05.993573558 +0000 UTC m=+0.614962166 container remove cfa57fb1289017d96a4a6115f4a5649dab1314f6605d07b5626b14096e94157f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-11T22:59:06.560 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:06 smithi131 podman[42477]: 2024-03-11 22:59:06.302803742 +0000 UTC m=+0.106691676 container create 3b8d8e27b2e71c285473e61fd55589a1b5fd452c3b40e32a196800a037558708 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-03-11T22:59:06.560 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:06 smithi131 podman[42477]: 2024-03-11 22:59:06.213846652 +0000 UTC m=+0.017734594 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:59:06.560 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:06 smithi131 podman[42477]: 2024-03-11 22:59:06.490522723 +0000 UTC m=+0.294410656 container init 3b8d8e27b2e71c285473e61fd55589a1b5fd452c3b40e32a196800a037558708 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-11T22:59:06.560 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:06 smithi131 podman[42477]: 2024-03-11 22:59:06.495781123 +0000 UTC m=+0.299669064 container start 3b8d8e27b2e71c285473e61fd55589a1b5fd452c3b40e32a196800a037558708 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-11T22:59:07.059 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:06 smithi131 podman[42477]: 2024-03-11 22:59:06.572089895 +0000 UTC m=+0.375977846 container attach 3b8d8e27b2e71c285473e61fd55589a1b5fd452c3b40e32a196800a037558708 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:59:07.060 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:06 smithi131 podman[42477]: 2024-03-11 22:59:06.817614753 +0000 UTC m=+0.621502689 container died 3b8d8e27b2e71c285473e61fd55589a1b5fd452c3b40e32a196800a037558708 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T22:59:07.363 INFO:journalctl@ceph.osd.1.smithi131.stdout:Mar 11 22:59:07 smithi131 podman[42625]: 2024-03-11 22:59:07.342219205 +0000 UTC m=+0.518800144 container remove 3b8d8e27b2e71c285473e61fd55589a1b5fd452c3b40e32a196800a037558708 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-94a5397) 2024-03-11T22:59:07.368 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T22:59:07.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:59:07.410 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T22:59:07.410 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T22:59:07.410 DEBUG:teuthology.orchestra.run.smithi195:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.2 2024-03-11T22:59:07.697 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:07 smithi195 systemd[1]: Stopping Ceph osd.2 for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:08.078 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:07 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2[34314]: 2024-03-11T22:59:07.924+0000 7f4f9f816700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:59:08.078 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:07 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2[34314]: 2024-03-11T22:59:07.924+0000 7f4f9f816700 -1 osd.2 31 *** Got signal Terminated *** 2024-03-11T22:59:08.078 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:07 smithi195 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2[34314]: 2024-03-11T22:59:07.924+0000 7f4f9f816700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T22:59:13.578 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:13 smithi195 podman[41961]: 2024-03-11 22:59:13.313290304 +0000 UTC m=+5.482626686 container died 23bc1d832960292c0d855238d2a16944a06c6234627dd141f1a733271408cfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T22:59:14.078 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:13 smithi195 podman[41961]: 2024-03-11 22:59:13.632528418 +0000 UTC m=+5.801864786 container cleanup 23bc1d832960292c0d855238d2a16944a06c6234627dd141f1a733271408cfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-94a5397, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-11T22:59:14.078 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:13 smithi195 bash[41961]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2 2024-03-11T22:59:14.078 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:13 smithi195 podman[42201]: 2024-03-11 22:59:13.825237394 +0000 UTC m=+0.515103498 container remove 23bc1d832960292c0d855238d2a16944a06c6234627dd141f1a733271408cfbd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True) 2024-03-11T22:59:14.539 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:14 smithi195 podman[42297]: 2024-03-11 22:59:14.144757773 +0000 UTC m=+0.018085214 image pull a6477a474d71b439ad6caa1ffdec0c8e4b844b029f7201a2e5a0a3cf1e1a739e quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b 2024-03-11T22:59:14.540 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:14 smithi195 podman[42297]: 2024-03-11 22:59:14.250994528 +0000 UTC m=+0.124322004 container create f1938ab3307ae8b9ea9d9e1e13af6c412739817d599049e998ccbd17e9074399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:59:14.540 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:14 smithi195 podman[42297]: 2024-03-11 22:59:14.45330715 +0000 UTC m=+0.326634594 container init f1938ab3307ae8b9ea9d9e1e13af6c412739817d599049e998ccbd17e9074399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T22:59:14.540 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:14 smithi195 podman[42297]: 2024-03-11 22:59:14.459281862 +0000 UTC m=+0.332609297 container start f1938ab3307ae8b9ea9d9e1e13af6c412739817d599049e998ccbd17e9074399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-94a5397, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T22:59:14.540 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:14 smithi195 podman[42297]: 2024-03-11 22:59:14.536145697 +0000 UTC m=+0.409473140 container attach f1938ab3307ae8b9ea9d9e1e13af6c412739817d599049e998ccbd17e9074399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-94a5397, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T22:59:14.828 INFO:journalctl@ceph.osd.2.smithi195.stdout:Mar 11 22:59:14 smithi195 podman[42297]: 2024-03-11 22:59:14.753784402 +0000 UTC m=+0.627111830 container died f1938ab3307ae8b9ea9d9e1e13af6c412739817d599049e998ccbd17e9074399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-94a5397, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T22:59:15.246 DEBUG:teuthology.orchestra.run.smithi195:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T22:59:15.286 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:59:15.287 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T22:59:15.287 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T22:59:15.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T22:59:15.587 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:59:15 smithi107 systemd[1]: Stopping Ceph rgw.foo.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:15.951 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:59:15 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a[41370]: 2024-03-11T22:59:15.582+0000 7f8cee9d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T22:59:15.951 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:59:15 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a[41370]: 2024-03-11T22:59:15.583+0000 7f8cfd249ac0 -1 shutting down 2024-03-11T22:59:25.879 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:59:25 smithi107 bash[55792]: time="2024-03-11T22:59:25Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T22:59:25.879 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:59:25 smithi107 podman[55792]: 2024-03-11 22:59:25.603402356 +0000 UTC m=+10.127105646 container died 998ec625860a8c61e7e2793e1a5daeaecb756b45c9e32b41801700368b56efae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-94a5397, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T22:59:26.201 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:59:25 smithi107 podman[55792]: 2024-03-11 22:59:25.957027028 +0000 UTC m=+10.480730322 container cleanup 998ec625860a8c61e7e2793e1a5daeaecb756b45c9e32b41801700368b56efae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:bbe755e768640cafd345af8384e9f3b099ff2fb85ebf0601e0883dbf412b887b, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-94a5397, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T22:59:26.202 INFO:journalctl@ceph.rgw.foo.a.smithi107.stdout:Mar 11 22:59:25 smithi107 bash[55792]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T22:59:26.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T22:59:26.446 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:59:26.447 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T22:59:26.447 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T22:59:26.447 DEBUG:teuthology.orchestra.run.smithi131:> sudo systemctl stop ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T22:59:26.731 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 systemd[1]: Stopping Ceph prometheus.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:27.029 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.727Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a[41297]: ts=2024-03-11T22:59:26.728Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T22:59:27.030 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:26 smithi131 podman[43357]: 2024-03-11 22:59:26.738511038 +0000 UTC m=+0.115644704 container died 164f3903a7a4adabf76d4a357861bb27f04aadc980866b16dbee2033f0d40f9f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:27.310 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:27 smithi131 podman[43357]: 2024-03-11 22:59:27.124879584 +0000 UTC m=+0.502013256 container cleanup 164f3903a7a4adabf76d4a357861bb27f04aadc980866b16dbee2033f0d40f9f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:27.311 INFO:journalctl@ceph.prometheus.a.smithi131.stdout:Mar 11 22:59:27 smithi131 bash[43357]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T22:59:27.470 DEBUG:teuthology.orchestra.run.smithi131:> sudo pkill -f 'journalctl -f -n 0 -u ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T22:59:27.508 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T22:59:27.508 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T22:59:27.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:59:27.694 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:59:29.202 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:29 smithi107 systemd[1]: Stopping Ceph alertmanager.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:29.619 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:29 smithi107 ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a[53275]: ts=2024-03-11T22:59:29.343Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T22:59:29.619 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:29 smithi107 podman[56767]: 2024-03-11 22:59:29.364166028 +0000 UTC m=+0.114242574 container died 88d11f1026ecbd1c8cb4c66b8ecd806b3a3ff524292839e137407aecb38e7348 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:29.951 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:29 smithi107 podman[56767]: 2024-03-11 22:59:29.807198173 +0000 UTC m=+0.557274725 container cleanup 88d11f1026ecbd1c8cb4c66b8ecd806b3a3ff524292839e137407aecb38e7348 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:29.951 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:29 smithi107 bash[56767]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T22:59:30.361 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:30 smithi107 podman[56786]: 2024-03-11 22:59:30.091724852 +0000 UTC m=+0.730675008 container remove 88d11f1026ecbd1c8cb4c66b8ecd806b3a3ff524292839e137407aecb38e7348 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:30.361 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:30 smithi107 podman[56786]: 2024-03-11 22:59:30.175480551 +0000 UTC m=+0.814430660 volume remove 53febce83a6e223c06d87ded58ce16e964293a66a83bec398e312664dd2943b1 2024-03-11T22:59:30.702 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:30 smithi107 systemd[1]: Stopping Ceph node-exporter.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:30.702 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:30 smithi107 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T22:59:30.702 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:30 smithi107 systemd[1]: Stopped Ceph alertmanager.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:59:30.702 INFO:journalctl@ceph.alertmanager.a.smithi107.stdout:Mar 11 22:59:30 smithi107 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.015s CPU time. 2024-03-11T22:59:31.191 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:30 smithi107 podman[56902]: 2024-03-11 22:59:30.859161691 +0000 UTC m=+0.148561083 container died 8bbd910ee2f1178d7873282f7c393dbe21952f7edf1c0ed5d5f5b16032ac706a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:31.191 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 podman[56902]: 2024-03-11 22:59:31.024251039 +0000 UTC m=+0.313650519 container exec_died 8bbd910ee2f1178d7873282f7c393dbe21952f7edf1c0ed5d5f5b16032ac706a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:31.451 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 podman[56902]: 2024-03-11 22:59:31.370192553 +0000 UTC m=+0.659591944 container cleanup 8bbd910ee2f1178d7873282f7c393dbe21952f7edf1c0ed5d5f5b16032ac706a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:31.451 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 bash[56902]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T22:59:31.951 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 podman[56921]: 2024-03-11 22:59:31.604807533 +0000 UTC m=+0.747818382 container remove 8bbd910ee2f1178d7873282f7c393dbe21952f7edf1c0ed5d5f5b16032ac706a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T22:59:31.951 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:59:31.951 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T22:59:31.951 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 systemd[1]: Stopped Ceph node-exporter.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:59:31.951 INFO:journalctl@ceph.node-exporter.a.smithi107.stdout:Mar 11 22:59:31 smithi107 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 1.920s CPU time. 2024-03-11T22:59:32.461 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:59:32.652 INFO:teuthology.orchestra.run.smithi131.stdout:Deleting cluster with fsid: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:59:34.060 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:33 smithi131 systemd[1]: Stopping Ceph grafana.a for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:34.342 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 podman[43750]: 2024-03-11 22:59:34.060055695 +0000 UTC m=+0.139126651 container died c5739fa709700d2b7f3f27bd753c5b6d1ea92c3d72b050909bf72f7a98bdad2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, distribution-scope=public, vendor=Red Hat, Inc., release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, build-date=2022-04-29T11:48:11.055897, version=8.5, maintainer=Paul Cuzner , architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-11T22:59:34.667 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 podman[43750]: 2024-03-11 22:59:34.412706146 +0000 UTC m=+0.491777099 container cleanup c5739fa709700d2b7f3f27bd753c5b6d1ea92c3d72b050909bf72f7a98bdad2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a, vcs-type=git, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, version=8.5, maintainer=Paul Cuzner , name=ubi8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=Ceph Grafana Container, io.buildah.version=1.32.0) 2024-03-11T22:59:34.667 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 bash[43750]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a 2024-03-11T22:59:35.052 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 podman[43769]: 2024-03-11 22:59:34.663780714 +0000 UTC m=+0.606625434 container remove c5739fa709700d2b7f3f27bd753c5b6d1ea92c3d72b050909bf72f7a98bdad2f (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-grafana-a, vendor=Red Hat, Inc., version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, vcs-type=git, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-11T22:59:35.052 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:59:35.052 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T22:59:35.052 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 systemd[1]: Stopped Ceph grafana.a for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:59:35.053 INFO:journalctl@ceph.grafana.a.smithi131.stdout:Mar 11 22:59:34 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@grafana.a.service: Consumed 3.915s CPU time. 2024-03-11T22:59:35.308 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:35 smithi131 systemd[1]: Stopping Ceph node-exporter.b for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:35.559 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:35 smithi131 podman[43888]: 2024-03-11 22:59:35.305462805 +0000 UTC m=+0.119077425 container died 4fa644216e24f35033fc0e27f08579ecdff4e8792166a1269e4bcb836d78942b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:59:36.050 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:35 smithi131 podman[43888]: 2024-03-11 22:59:35.674747539 +0000 UTC m=+0.488362151 container cleanup 4fa644216e24f35033fc0e27f08579ecdff4e8792166a1269e4bcb836d78942b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:59:36.050 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:35 smithi131 bash[43888]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T22:59:36.050 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:35 smithi131 podman[43907]: 2024-03-11 22:59:35.851062873 +0000 UTC m=+0.547825439 container remove 4fa644216e24f35033fc0e27f08579ecdff4e8792166a1269e4bcb836d78942b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T22:59:36.050 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:35 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:59:36.349 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:36 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T22:59:36.350 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:36 smithi131 systemd[1]: Stopped Ceph node-exporter.b for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:59:36.350 INFO:journalctl@ceph.node-exporter.b.smithi131.stdout:Mar 11 22:59:36 smithi131 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 1.840s CPU time. 2024-03-11T22:59:36.704 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T22:59:36.898 INFO:teuthology.orchestra.run.smithi195.stdout:Deleting cluster with fsid: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:59:37.828 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:37 smithi195 systemd[1]: Stopping Ceph node-exporter.c for eb9eeed0-dff9-11ee-95c8-87774f69a715... 2024-03-11T22:59:38.290 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:37 smithi195 podman[43140]: 2024-03-11 22:59:37.992981059 +0000 UTC m=+0.128768875 container died 1f0bf101b8efc561f701dfbc756e59577253ad98df2363e4939544a58611352f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:59:38.291 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:38 smithi195 podman[43140]: 2024-03-11 22:59:38.287441538 +0000 UTC m=+0.423229345 container cleanup 1f0bf101b8efc561f701dfbc756e59577253ad98df2363e4939544a58611352f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:59:38.578 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:38 smithi195 bash[43140]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c 2024-03-11T22:59:38.578 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:38 smithi195 podman[43159]: 2024-03-11 22:59:38.496793921 +0000 UTC m=+0.504415804 container remove 1f0bf101b8efc561f701dfbc756e59577253ad98df2363e4939544a58611352f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715-node-exporter-c, maintainer=The Prometheus Authors ) 2024-03-11T22:59:38.578 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:38 smithi195 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.c.service: Main process exited, code=exited, status=143/n/a 2024-03-11T22:59:38.887 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:38 smithi195 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.c.service: Failed with result 'exit-code'. 2024-03-11T22:59:38.887 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:38 smithi195 systemd[1]: Stopped Ceph node-exporter.c for eb9eeed0-dff9-11ee-95c8-87774f69a715. 2024-03-11T22:59:38.887 INFO:journalctl@ceph.node-exporter.c.smithi195.stdout:Mar 11 22:59:38 smithi195 systemd[1]: ceph-eb9eeed0-dff9-11ee-95c8-87774f69a715@node-exporter.c.service: Consumed 1.846s CPU time. 2024-03-11T22:59:39.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:59:39.323 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:59:39.359 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T22:59:39.389 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T22:59:39.391 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi107/crash 2024-03-11T22:59:39.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:59:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:59:39.441 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:59:39.444 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi131/crash 2024-03-11T22:59:39.445 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:59:39.491 INFO:teuthology.orchestra.run.smithi131.stderr:tar: /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:59:39.491 INFO:teuthology.orchestra.run.smithi131.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:59:39.493 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi195/crash 2024-03-11T22:59:39.494 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash -- . 2024-03-11T22:59:39.537 INFO:teuthology.orchestra.run.smithi195.stderr:tar: /var/lib/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T22:59:39.537 INFO:teuthology.orchestra.run.smithi195.stderr:tar: Error is not recoverable: exiting now 2024-03-11T22:59:39.538 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T22:59:39.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:59:39.571 INFO:teuthology.orchestra.run.smithi107.stdout:2024-03-11T22:55:17.718649+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:59:39.571 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T22:59:39.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[SEC\]' /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:59:39.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]' /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:59:39.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[WRN\]' /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-11T22:59:39.779 INFO:teuthology.orchestra.run.smithi107.stdout:2024-03-11T22:55:17.718649+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-11T22:59:39.780 INFO:tasks.cephadm:Compressing logs... 2024-03-11T22:59:39.780 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:59:39.822 DEBUG:teuthology.orchestra.run.smithi131:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:59:39.825 DEBUG:teuthology.orchestra.run.smithi195:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T22:59:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T22:59:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:59:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T22:59:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:59:39.847 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr: 57.7% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.y.log: /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr: 82.8% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log: 90.1% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log: 85.0% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:59:39.851 INFO:teuthology.orchestra.run.smithi107.stderr: 83.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:59:39.855 INFO:teuthology.orchestra.run.smithi195.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T22:59:39.855 INFO:teuthology.orchestra.run.smithi195.stderr:: No such file or directory 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.z.log 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log: /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr: 81.7% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log: 90.2% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log: 84.7% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:59:39.856 INFO:teuthology.orchestra.run.smithi195.stderr: 78.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:59:39.858 INFO:teuthology.orchestra.run.smithi131.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose: No such file or directory -- 2024-03-11T22:59:39.858 INFO:teuthology.orchestra.run.smithi131.stderr: /var/log/ceph/cephadm.log 2024-03-11T22:59:39.858 INFO:teuthology.orchestra.run.smithi131.stderr:gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T22:59:39.858 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T22:59:39.858 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T22:59:39.858 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T22:59:39.860 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log 2024-03-11T22:59:39.860 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T22:59:39.860 INFO:teuthology.orchestra.run.smithi131.stderr: 90.2% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T22:59:39.860 INFO:teuthology.orchestra.run.smithi131.stderr: 81.6% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T22:59:39.861 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T22:59:39.861 INFO:teuthology.orchestra.run.smithi131.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.b.log: 84.6% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T22:59:39.861 INFO:teuthology.orchestra.run.smithi131.stderr: 77.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T22:59:39.861 INFO:teuthology.orchestra.run.smithi131.stderr: 90.3% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T22:59:39.862 INFO:teuthology.orchestra.run.smithi195.stderr:/var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.c.log: 90.3% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.z.log.gz 2024-03-11T22:59:39.871 INFO:teuthology.orchestra.run.smithi195.stderr: 90.3% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:59:39.875 INFO:teuthology.orchestra.run.smithi107.stderr: 90.2% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:59:39.881 INFO:teuthology.orchestra.run.smithi131.stderr: 90.2% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T22:59:39.925 INFO:teuthology.orchestra.run.smithi195.stderr: 92.5% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T22:59:39.925 INFO:teuthology.orchestra.run.smithi131.stderr: 92.4% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T22:59:39.948 INFO:teuthology.orchestra.run.smithi107.stderr: 88.1% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T22:59:40.077 INFO:teuthology.orchestra.run.smithi107.stderr: 91.6% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T22:59:40.128 INFO:teuthology.orchestra.run.smithi107.stderr: 94.3% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T22:59:40.130 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-03-11T22:59:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m0.296s 2024-03-11T22:59:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m0.624s 2024-03-11T22:59:40.130 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.049s 2024-03-11T22:59:40.130 INFO:teuthology.orchestra.run.smithi195.stderr: 94.3% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T22:59:40.131 INFO:teuthology.orchestra.run.smithi195.stderr: 2024-03-11T22:59:40.131 INFO:teuthology.orchestra.run.smithi195.stderr:real 0m0.291s 2024-03-11T22:59:40.131 INFO:teuthology.orchestra.run.smithi195.stderr:user 0m0.369s 2024-03-11T22:59:40.131 INFO:teuthology.orchestra.run.smithi195.stderr:sys 0m0.033s 2024-03-11T22:59:40.181 INFO:teuthology.orchestra.run.smithi131.stderr: 94.3% -- replaced with /var/log/ceph/eb9eeed0-dff9-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T22:59:40.182 INFO:teuthology.orchestra.run.smithi131.stderr: 2024-03-11T22:59:40.183 INFO:teuthology.orchestra.run.smithi131.stderr:real 0m0.342s 2024-03-11T22:59:40.183 INFO:teuthology.orchestra.run.smithi131.stderr:user 0m0.429s 2024-03-11T22:59:40.183 INFO:teuthology.orchestra.run.smithi131.stderr:sys 0m0.036s 2024-03-11T22:59:40.183 INFO:tasks.cephadm:Archiving logs... 2024-03-11T22:59:40.183 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi107/log 2024-03-11T22:59:40.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:59:40.325 DEBUG:teuthology.misc:Transferring archived files from smithi131:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi131/log 2024-03-11T22:59:40.326 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:59:40.427 DEBUG:teuthology.misc:Transferring archived files from smithi195:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi195/log 2024-03-11T22:59:40.427 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:59:40.520 INFO:tasks.cephadm:Removing cluster... 2024-03-11T22:59:40.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 --force 2024-03-11T22:59:40.703 INFO:teuthology.orchestra.run.smithi107.stdout:Deleting cluster with fsid: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:59:40.821 DEBUG:teuthology.orchestra.run.smithi131:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 --force 2024-03-11T22:59:41.008 INFO:teuthology.orchestra.run.smithi131.stdout:Deleting cluster with fsid: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:59:41.123 DEBUG:teuthology.orchestra.run.smithi195:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid eb9eeed0-dff9-11ee-95c8-87774f69a715 --force 2024-03-11T22:59:41.303 INFO:teuthology.orchestra.run.smithi195.stdout:Deleting cluster with fsid: eb9eeed0-dff9-11ee-95c8-87774f69a715 2024-03-11T22:59:41.419 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T22:59:41.419 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:59:41.435 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:59:41.451 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:59:41.466 INFO:tasks.cephadm:Teardown complete 2024-03-11T22:59:41.467 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-03-11T22:59:41.478 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi107:/dev/vg_nvme/lv_1... 2024-03-11T22:59:41.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo nvme disconnect -n lv_1 2024-03-11T22:59:41.613 INFO:teuthology.orchestra.run.smithi107.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:59:41.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:41.615 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi107:/dev/vg_nvme/lv_2... 2024-03-11T22:59:41.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo nvme disconnect -n lv_2 2024-03-11T22:59:41.759 INFO:teuthology.orchestra.run.smithi107.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:59:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:41.760 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi107:/dev/vg_nvme/lv_3... 2024-03-11T22:59:41.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo nvme disconnect -n lv_3 2024-03-11T22:59:41.893 INFO:teuthology.orchestra.run.smithi107.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:59:41.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:41.894 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi107:/dev/vg_nvme/lv_4... 2024-03-11T22:59:41.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo nvme disconnect -n lv_4 2024-03-11T22:59:42.046 INFO:teuthology.orchestra.run.smithi107.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:59:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:42.048 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-03-11T22:59:42.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/scratch_devs 2024-03-11T22:59:42.076 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_1... 2024-03-11T22:59:42.077 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_1 2024-03-11T22:59:42.233 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:59:42.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:42.235 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_2... 2024-03-11T22:59:42.235 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_2 2024-03-11T22:59:42.386 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:59:42.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:42.388 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_3... 2024-03-11T22:59:42.388 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_3 2024-03-11T22:59:42.537 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:59:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:42.538 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi131:/dev/vg_nvme/lv_4... 2024-03-11T22:59:42.539 DEBUG:teuthology.orchestra.run.smithi131:> sudo nvme disconnect -n lv_4 2024-03-11T22:59:42.680 INFO:teuthology.orchestra.run.smithi131.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:59:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:42.681 DEBUG:teuthology.orchestra.run.smithi131:> set -ex 2024-03-11T22:59:42.682 DEBUG:teuthology.orchestra.run.smithi131:> sudo dd of=/scratch_devs 2024-03-11T22:59:42.711 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi195:/dev/vg_nvme/lv_1... 2024-03-11T22:59:42.711 DEBUG:teuthology.orchestra.run.smithi195:> sudo nvme disconnect -n lv_1 2024-03-11T22:59:42.856 INFO:teuthology.orchestra.run.smithi195.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-03-11T22:59:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:42.858 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi195:/dev/vg_nvme/lv_2... 2024-03-11T22:59:42.858 DEBUG:teuthology.orchestra.run.smithi195:> sudo nvme disconnect -n lv_2 2024-03-11T22:59:42.994 INFO:teuthology.orchestra.run.smithi195.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-03-11T22:59:42.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:42.996 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi195:/dev/vg_nvme/lv_3... 2024-03-11T22:59:42.996 DEBUG:teuthology.orchestra.run.smithi195:> sudo nvme disconnect -n lv_3 2024-03-11T22:59:43.160 INFO:teuthology.orchestra.run.smithi195.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-03-11T22:59:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:43.161 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi195:/dev/vg_nvme/lv_4... 2024-03-11T22:59:43.162 DEBUG:teuthology.orchestra.run.smithi195:> sudo nvme disconnect -n lv_4 2024-03-11T22:59:43.330 INFO:teuthology.orchestra.run.smithi195.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-03-11T22:59:43.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:43.331 DEBUG:teuthology.orchestra.run.smithi195:> set -ex 2024-03-11T22:59:43.332 DEBUG:teuthology.orchestra.run.smithi195:> sudo dd of=/scratch_devs 2024-03-11T22:59:43.360 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T22:59:43.370 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T22:59:43.371 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:59:43.375 DEBUG:teuthology.orchestra.run.smithi131:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:59:43.377 DEBUG:teuthology.orchestra.run.smithi195:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:59:43.389 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:59:43.392 INFO:teuthology.orchestra.run.smithi131.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:59:43.393 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:59:43.394 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-03-11T22:59:43.394 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +3772us[+3772us] +/- 134ms 2024-03-11T22:59:43.394 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -1235us[-1238us] +/- 72ms 2024-03-11T22:59:43.394 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -780us[ -783us] +/- 80ms 2024-03-11T22:59:43.394 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:59:43.397 INFO:teuthology.orchestra.run.smithi131.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:59:43.397 INFO:teuthology.orchestra.run.smithi131.stdout:=============================================================================== 2024-03-11T22:59:43.397 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +3776us[+3776us] +/- 134ms 2024-03-11T22:59:43.397 INFO:teuthology.orchestra.run.smithi131.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -1221us[-1214us] +/- 72ms 2024-03-11T22:59:43.398 INFO:teuthology.orchestra.run.smithi131.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 -770us[ -770us] +/- 80ms 2024-03-11T22:59:43.398 INFO:teuthology.orchestra.run.smithi131.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:59:43.414 INFO:teuthology.orchestra.run.smithi195.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:59:43.418 INFO:teuthology.orchestra.run.smithi195.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:59:43.418 INFO:teuthology.orchestra.run.smithi195.stdout:=============================================================================== 2024-03-11T22:59:43.418 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +3771us[+3771us] +/- 134ms 2024-03-11T22:59:43.418 INFO:teuthology.orchestra.run.smithi195.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -1225us[-1210us] +/- 72ms 2024-03-11T22:59:43.418 INFO:teuthology.orchestra.run.smithi195.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -761us[ -761us] +/- 80ms 2024-03-11T22:59:43.418 INFO:teuthology.orchestra.run.smithi195.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:59:43.419 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T22:59:43.431 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T22:59:43.432 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T22:59:43.442 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:59:43.522 DEBUG:teuthology.orchestra.run.smithi131:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:59:43.598 DEBUG:teuthology.orchestra.run.smithi195:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:59:43.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:59:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:43.699 DEBUG:teuthology.orchestra.run.smithi131:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:59:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:43.731 DEBUG:teuthology.orchestra.run.smithi195:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-11T22:59:43.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:43.761 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T22:59:43.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T22:59:43.781 INFO:teuthology.task.internal:Duration was 750.562050 seconds 2024-03-11T22:59:43.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T22:59:43.791 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T22:59:43.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:59:43.794 DEBUG:teuthology.orchestra.run.smithi131:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:59:43.797 DEBUG:teuthology.orchestra.run.smithi195:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:59:43.839 INFO:teuthology.orchestra.run.smithi131.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:59:43.839 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:59:43.843 INFO:teuthology.orchestra.run.smithi195.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:59:44.364 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T22:59:44.364 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-03-11T22:59:44.364 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:59:44.387 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi131.front.sepia.ceph.com 2024-03-11T22:59:44.387 DEBUG:teuthology.orchestra.run.smithi131:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:59:44.408 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi195.front.sepia.ceph.com 2024-03-11T22:59:44.408 DEBUG:teuthology.orchestra.run.smithi195:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:59:44.441 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T22:59:44.442 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:59:44.444 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:59:44.450 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:59:44.510 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T22:59:44.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:59:44.516 DEBUG:teuthology.orchestra.run.smithi131:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:59:44.518 DEBUG:teuthology.orchestra.run.smithi195:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:59:44.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T22:59:44.720 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T22:59:44.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:59:44.748 DEBUG:teuthology.orchestra.run.smithi131:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:59:44.776 DEBUG:teuthology.orchestra.run.smithi195:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:59:44.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T22:59:44.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:59:44.818 DEBUG:teuthology.orchestra.run.smithi131:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:59:44.820 DEBUG:teuthology.orchestra.run.smithi195:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:59:44.841 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-03-11T22:59:44.843 INFO:teuthology.orchestra.run.smithi131.stdout:kernel.core_pattern = core 2024-03-11T22:59:44.869 INFO:teuthology.orchestra.run.smithi195.stdout:kernel.core_pattern = core 2024-03-11T22:59:44.896 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:59:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:44.920 DEBUG:teuthology.orchestra.run.smithi131:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:59:44.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:44.934 DEBUG:teuthology.orchestra.run.smithi195:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:59:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:59:44.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T22:59:44.966 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T22:59:44.967 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi107 2024-03-11T22:59:44.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:59:45.006 DEBUG:teuthology.misc:Transferring archived files from smithi131:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi131 2024-03-11T22:59:45.007 DEBUG:teuthology.orchestra.run.smithi131:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:59:45.043 DEBUG:teuthology.misc:Transferring archived files from smithi195:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032/remote/smithi195 2024-03-11T22:59:45.044 DEBUG:teuthology.orchestra.run.smithi195:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:59:45.080 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T22:59:45.080 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:59:45.082 DEBUG:teuthology.orchestra.run.smithi131:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:59:45.084 DEBUG:teuthology.orchestra.run.smithi195:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:59:45.130 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T22:59:45.143 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T22:59:45.143 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T22:59:45.156 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T22:59:45.156 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:59:45.159 DEBUG:teuthology.orchestra.run.smithi131:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:59:45.161 DEBUG:teuthology.orchestra.run.smithi195:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:59:45.171 INFO:teuthology.orchestra.run.smithi107.stdout: 265936 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:59 /home/ubuntu/cephtest 2024-03-11T22:59:45.174 INFO:teuthology.orchestra.run.smithi131.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:59 /home/ubuntu/cephtest 2024-03-11T22:59:45.184 INFO:teuthology.orchestra.run.smithi195.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:59 /home/ubuntu/cephtest 2024-03-11T22:59:45.186 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T22:59:45.199 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T22:59:45.229 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T22:59:45.258 DEBUG:teuthology.nuke:shortname: smithi107 2024-03-11T22:59:45.258 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:59:45.284 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:37:42.507855', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLSqpIXwqoo3HpDn9TD0OOJwViOUqXSLR4yiFNtWg4Kr2m4I23H49N2+5gXmUp+v6RaPTVUbM+uvFLGNfQTtr0Q='} 2024-03-11T22:59:45.289 DEBUG:teuthology.nuke:shortname: smithi131 2024-03-11T22:59:45.289 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:59:45.313 DEBUG:teuthology.nuke:shortname: smithi195 2024-03-11T22:59:45.313 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:59:45.318 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi131.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:37:42.506070', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY2Rv8Cyuf2grkHYrJp+swE8nONHUG3IwxKX2IISZcByZzxRAZWcqRfUz4dJcBHKVZGYM/LiOZL72omdtxaiQs='} 2024-03-11T22:59:45.338 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi195.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_21:08:02-orch-squid-distro-default-smithi/7592032', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 22:37:42.508815', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJbFDz/gnTug6cnYTXefQ/S1iqAwKM7AOLnpzvSHd54VEHPsn8szxjYLrvDh229KOl1kWeyHZs2fjcy7BROhulM='} 2024-03-11T22:59:45.350 INFO:teuthology.orchestra.console.smithi107:Power off 2024-03-11T22:59:45.351 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:59:45.376 DEBUG:teuthology.orchestra.console.smithi107:power off output: Chassis Power Control: Down/Off 2024-03-11T22:59:45.377 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:45.393 INFO:teuthology.orchestra.console.smithi131:Power off 2024-03-11T22:59:45.394 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:59:45.408 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-11T22:59:45.419 DEBUG:teuthology.orchestra.console.smithi131:power off output: Chassis Power Control: Down/Off 2024-03-11T22:59:45.419 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:45.438 INFO:teuthology.orchestra.console.smithi195:Power off 2024-03-11T22:59:45.438 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:59:45.448 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-03-11T22:59:45.461 DEBUG:teuthology.orchestra.console.smithi195:power off output: Chassis Power Control: Down/Off 2024-03-11T22:59:45.461 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:45.486 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-11T22:59:49.410 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:49.449 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:49.487 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:49.519 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-11T22:59:49.559 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is on 2024-03-11T22:59:49.597 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is on 2024-03-11T22:59:53.522 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:53.560 DEBUG:teuthology.orchestra.console.smithi131:pexpect command: ipmitool -H smithi131.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:53.598 DEBUG:teuthology.orchestra.console.smithi195:pexpect command: ipmitool -H smithi195.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:53.633 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is on 2024-03-11T22:59:53.670 DEBUG:teuthology.orchestra.console.smithi131:check power output: Chassis Power is off 2024-03-11T22:59:53.707 DEBUG:teuthology.orchestra.console.smithi195:check power output: Chassis Power is off 2024-03-11T22:59:53.772 INFO:teuthology.orchestra.console.smithi131:Power off completed 2024-03-11T22:59:53.808 INFO:teuthology.orchestra.console.smithi195:Power off completed 2024-03-11T22:59:57.638 DEBUG:teuthology.orchestra.console.smithi107:pexpect command: ipmitool -H smithi107.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:59:57.748 DEBUG:teuthology.orchestra.console.smithi107:check power output: Chassis Power is off 2024-03-11T22:59:57.849 INFO:teuthology.orchestra.console.smithi107:Power off completed 2024-03-11T22:59:57.953 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/on fixed-2 mon_election/classic start} duration: 750.5620503425598 failure_reason: '"2024-03-11T22:55:17.718649+0000 mon.a (mon.0) 247 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-11T22:59:57.953 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:59:58.037 INFO:teuthology.run:FAIL