2024-02-10T05:28:43.725 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-10T05:28:43.726 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} --verbose --name yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085 -- /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/orig.config.yaml 2024-02-10T05:28:43.754 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:28:43.854 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085 branch: main description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} email: yweinste@redhat.com first_in_suite: false job_id: '7555085' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 1 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.1 - client.1 - prometheus.a - grafana.a - node-exporter.b - - mon.c - mgr.z - osd.2 - client.2 - node-exporter.c seed: 1610 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFhCtJsG1Y7KtIGGRFYs/Qw4Gb3bJ/kDTOlam96Eze/TkWzv7QHpVzPR03nV5L/q69PJ+vRyTzPEGJpmeIVJPqk= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPyIf7TFhaSSvNFZNfHVTK2G3PwGZiyzwCnv2LgKRRQazhS5CvY3ki6HYcsuE4tQSRsVdPAA/pl1hA+ZU837MLE= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM55KO4V6S5wHn97KAIpq4FEXb4kwKujPIjWqsg/Rv4afevy8tyVREpmrfuCa+L3ZIhsV3gc15G9VjKPB6Qg6L0= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - nvme_loop: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - cephadm.shell: mon.a: - stat -c '%u %g' /var/log/ceph | grep '167 167' - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph orch ls --format yaml - ceph orch ls | grep '^osd ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:44:16 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-10T05:28:43.854 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-10T05:28:43.855 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-10T05:28:43.855 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-10T05:28:43.857 INFO:teuthology.task.internal:Checking packages... 2024-02-10T05:28:43.893 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-10T05:28:43.894 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-10T05:28:43.894 INFO:teuthology.packaging:ref: None 2024-02-10T05:28:43.894 INFO:teuthology.packaging:tag: None 2024-02-10T05:28:43.894 INFO:teuthology.packaging:branch: main 2024-02-10T05:28:43.894 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:28:43.894 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-10T05:28:44.193 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-10T05:28:44.195 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-10T05:28:44.202 INFO:teuthology.task.internal:no buildpackages task found 2024-02-10T05:28:44.202 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-10T05:28:44.212 INFO:teuthology.task.internal:Saving configuration 2024-02-10T05:28:44.234 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-10T05:28:44.240 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:28:44.267 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:21:33.274214', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFhCtJsG1Y7KtIGGRFYs/Qw4Gb3bJ/kDTOlam96Eze/TkWzv7QHpVzPR03nV5L/q69PJ+vRyTzPEGJpmeIVJPqk='} 2024-02-10T05:28:44.292 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:21:33.272747', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPyIf7TFhaSSvNFZNfHVTK2G3PwGZiyzwCnv2LgKRRQazhS5CvY3ki6HYcsuE4tQSRsVdPAA/pl1hA+ZU837MLE='} 2024-02-10T05:28:44.316 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:21:33.275459', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM55KO4V6S5wHn97KAIpq4FEXb4kwKujPIjWqsg/Rv4afevy8tyVREpmrfuCa+L3ZIhsV3gc15G9VjKPB6Qg6L0='} 2024-02-10T05:28:44.317 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-10T05:28:44.322 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-02-10T05:28:44.323 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.1', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-02-10T05:28:44.323 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'osd.2', 'client.2', 'node-exporter.c'] 2024-02-10T05:28:44.323 INFO:teuthology.run_tasks:Running task console_log... 2024-02-10T05:28:44.427 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f89a835b160>, signals=[15]) 2024-02-10T05:28:44.428 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-10T05:28:44.436 INFO:teuthology.task.internal:Opening connections... 2024-02-10T05:28:44.436 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-02-10T05:28:44.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:28:44.541 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-02-10T05:28:44.542 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:28:44.633 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2024-02-10T05:28:44.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:28:44.736 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-10T05:28:44.749 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-02-10T05:28:44.768 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-02-10T05:28:44.768 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-02-10T05:28:44.826 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-02-10T05:28:44.826 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-02-10T05:28:44.826 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-02-10T05:28:44.826 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:28:44.826 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:28:44.827 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:28:44.828 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:28:44.828 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-02-10T05:28:44.854 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-02-10T05:28:44.872 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-02-10T05:28:44.872 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:28:44.930 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:28:44.931 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:28:44.931 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:28:44.931 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:28:44.931 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:28:44.931 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-02-10T05:28:44.945 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2024-02-10T05:28:44.963 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2024-02-10T05:28:44.964 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="9" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="9" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el9" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:LOGO="fedora-logo-icon" 2024-02-10T05:28:45.020 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-10T05:28:45.021 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2024-02-10T05:28:45.021 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-10T05:28:45.021 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-10T05:28:45.021 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-10T05:28:45.021 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2024-02-10T05:28:45.045 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-10T05:28:45.060 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-10T05:28:45.067 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-10T05:28:45.067 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:28:45.070 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:28:45.072 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2024-02-10T05:28:45.088 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-10T05:28:45.094 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-10T05:28:45.094 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:28:45.127 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:28:45.130 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2024-02-10T05:28:45.446 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-10T05:28:45.529 INFO:teuthology.run_tasks:Running task kernel... 2024-02-10T05:28:45.543 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-02-10T05:28:45.543 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-02-10T05:28:45.543 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:28:45.544 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:28:45.544 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-10T05:28:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:28:45.561 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-02-10T05:28:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:28:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:28:45.562 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-02-10T05:28:45.563 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-02-10T05:28:45.617 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:28:45.618 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.14.0-361.el9.x86_64 2024-02-10T05:28:45.618 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-02-10T05:28:45.619 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:28:45.619 INFO:teuthology.task.kernel:Running kernel on smithi110: 5.14.0-361.el9.x86_64 2024-02-10T05:28:45.619 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2024-02-10T05:28:45.620 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-02-10T05:28:45.620 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-02-10T05:28:45.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-02-10T05:28:47.643 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-02-10T05:28:47.719 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-10T05:28:48.745 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-02-10T05:28:49.268 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS-9 - BaseOS 3.9 MB/s | 2.0 MB 00:00 2024-02-10T05:28:49.370 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-02-10T05:28:50.196 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-02-10T05:28:50.810 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.0 MB 00:01 2024-02-10T05:28:51.705 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 6.9 MB/s | 8.0 MB 00:01 2024-02-10T05:28:52.259 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-02-10T05:28:53.999 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-02-10T05:28:54.011 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 19 MB 00:02 2024-02-10T05:28:56.360 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 19 MB 00:03 2024-02-10T05:28:58.370 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 5.9 MB 00:00 2024-02-10T05:28:59.114 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 5.9 MB 00:01 2024-02-10T05:29:00.296 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 15 kB 00:00 2024-02-10T05:29:00.987 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 15 kB 00:00 2024-02-10T05:29:03.135 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 5.9 MB 00:03 2024-02-10T05:29:04.603 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-02-10T05:29:04.683 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 4.8 MB/s | 20 MB 00:04 2024-02-10T05:29:09.580 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-02-10T05:29:11.837 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:29:11.837 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:29:11.933 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-02-10T05:29:11.944 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-02-10T05:29:11.944 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-02-10T05:29:11.944 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-02-10T05:29:11.944 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-02-10T05:29:11.945 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:29:11.946 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 99 M 2024-02-10T05:29:11.946 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 122 M 2024-02-10T05:29:11.946 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-02-10T05:29:11.979 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 2.8 MB/s | 20 MB 00:07 2024-02-10T05:29:12.348 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-02-10T05:29:12.799 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 24 MB/s | 21 MB 00:00 2024-02-10T05:29:13.674 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 23 MB/s | 39 MB 00:01 2024-02-10T05:29:13.974 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 21 MB/s | 33 MB 00:01 2024-02-10T05:29:13.977 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:29:13.978 INFO:teuthology.orchestra.run.smithi105.stdout:Total 49 MB/s | 99 MB 00:02 2024-02-10T05:29:13.978 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-02-10T05:29:14.354 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-02-10T05:29:14.354 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-02-10T05:29:15.377 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-02-10T05:29:15.378 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-02-10T05:29:16.770 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-02-10T05:29:17.803 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-02-10T05:29:18.475 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:29:18.705 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:29:19.068 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:29:19.069 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:29:19.177 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:29:19.184 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-02-10T05:29:19.185 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-02-10T05:29:19.185 INFO:teuthology.orchestra.run.smithi110.stdout:Install 4 Packages 2024-02-10T05:29:19.185 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:29:19.185 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 99 M 2024-02-10T05:29:19.185 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 122 M 2024-02-10T05:29:19.185 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-02-10T05:29:19.571 INFO:teuthology.orchestra.run.smithi110.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 16 MB/s | 6.0 MB 00:00 2024-02-10T05:29:20.138 INFO:teuthology.orchestra.run.smithi110.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-02-10T05:29:20.805 INFO:teuthology.orchestra.run.smithi110.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 27 MB/s | 33 MB 00:01 2024-02-10T05:29:21.147 INFO:teuthology.orchestra.run.smithi110.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 20 MB/s | 39 MB 00:01 2024-02-10T05:29:21.150 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:29:21.150 INFO:teuthology.orchestra.run.smithi110.stdout:Total 51 MB/s | 99 MB 00:01 2024-02-10T05:29:21.151 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-02-10T05:29:21.489 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:29:21.530 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-02-10T05:29:21.530 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-02-10T05:29:21.787 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:29:22.581 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-02-10T05:29:22.581 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-02-10T05:29:23.347 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 938 kB/s | 20 MB 00:22 2024-02-10T05:29:24.986 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-02-10T05:29:25.770 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:29:25.900 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:29:26.465 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:29:26.664 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:29:28.424 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-02-10T05:29:28.670 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:29:28.844 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:29:30.866 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-10T05:29:30.866 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-10T05:29:30.973 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-10T05:29:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-10T05:29:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:29:30.989 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-02-10T05:29:30.989 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T05:29:30.989 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-02-10T05:29:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:29:30.990 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 99 M 2024-02-10T05:29:30.990 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 122 M 2024-02-10T05:29:30.990 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-02-10T05:29:31.355 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:29:31.377 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 15 MB/s | 6.0 MB 00:00 2024-02-10T05:29:32.228 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 17 MB/s | 21 MB 00:01 2024-02-10T05:29:33.062 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-modules-core-5.14.0-419.el9.x86_6 20 MB/s | 33 MB 00:01 2024-02-10T05:29:33.568 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:29:33.757 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:29:33.787 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 14 MB/s | 39 MB 00:02 2024-02-10T05:29:33.790 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:29:33.790 INFO:teuthology.orchestra.run.smithi029.stdout:Total 35 MB/s | 99 MB 00:02 2024-02-10T05:29:33.791 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-02-10T05:29:34.286 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-02-10T05:29:34.286 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-02-10T05:29:35.332 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-02-10T05:29:35.333 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-02-10T05:29:37.890 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-02-10T05:29:38.507 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:29:38.704 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:29:38.901 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:29:41.798 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:29:42.086 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:29:47.181 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:29:47.338 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:29:52.458 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:00.290 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:00.291 INFO:teuthology.orchestra.run.smithi105.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:30:00.291 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:30:00.359 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:00.747 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:00.747 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:30:00.747 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:30:00.747 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:30:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:01.168 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:30:01.168 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-02-10T05:30:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:30:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:30:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:30:01.169 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:30:01.169 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-02-10T05:30:01.671 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:30:01.839 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:01.840 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:30:01.840 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:30:01.840 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:30:01.840 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:30:01.840 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-02-10T05:30:01.840 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:30:01.841 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:30:01.947 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:01.948 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:30:01.948 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2024-02-10T05:30:01.983 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2024-02-10T05:30:01.983 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2024-02-10T05:30:01.983 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.06 2024-02-10T05:30:01.983 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 68.el9 2024-02-10T05:30:01.983 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2024-02-10T05:30:01.983 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Group : Unspecified 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 8264254 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : builder@centos.org 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:30:01.984 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:30:01.985 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:30:01.985 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:30:01.985 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2024-02-10T05:30:01.985 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:30:01.985 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:30:01.986 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:30:01.986 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:30:01.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:30:02.660 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2024-02-10T05:30:04.096 INFO:teuthology.orchestra.run.smithi105.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:30:04.126 INFO:teuthology.orchestra.run.smithi105.stderr:done 2024-02-10T05:30:04.129 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-02-10T05:30:04.185 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.SKNMlpQdKs 2024-02-10T05:30:04.185 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SKNMlpQdKs 2024-02-10T05:30:04.259 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.SKNMlpQdKs 2024-02-10T05:30:04.403 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.SKNMlpQdKs is 6KB 2024-02-10T05:30:04.452 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.SKNMlpQdKs 2024-02-10T05:30:04.466 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:30:04.531 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:30:04.531 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:30:04.531 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:30:04.531 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:30:04.532 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:30:04.676 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2024-02-10T05:30:07.666 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:07.666 INFO:teuthology.orchestra.run.smithi110.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:30:07.666 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:30:07.737 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:08.098 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:08.099 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:30:08.099 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:30:08.099 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:30:08.520 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-02-10T05:30:09.034 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:30:09.156 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:09.156 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:30:09.156 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:30:09.156 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:30:09.156 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:30:09.156 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-02-10T05:30:09.156 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:30:09.157 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:30:09.264 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:09.264 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:30:09.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2024-02-10T05:30:09.299 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2024-02-10T05:30:09.299 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2024-02-10T05:30:09.299 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.06 2024-02-10T05:30:09.299 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 68.el9 2024-02-10T05:30:09.299 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Group : Unspecified 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 8264254 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : builder@centos.org 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2024-02-10T05:30:09.300 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2024-02-10T05:30:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:30:09.301 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:30:09.301 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:30:09.301 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:30:09.301 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2024-02-10T05:30:09.301 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:30:09.301 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:30:09.302 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:30:09.302 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:30:09.302 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:30:09.916 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2024-02-10T05:30:11.335 INFO:teuthology.orchestra.run.smithi110.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:30:11.376 INFO:teuthology.orchestra.run.smithi110.stderr:done 2024-02-10T05:30:11.379 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2024-02-10T05:30:11.409 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.iNsl9aBgoe 2024-02-10T05:30:11.409 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iNsl9aBgoe 2024-02-10T05:30:11.479 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.iNsl9aBgoe 2024-02-10T05:30:11.623 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.iNsl9aBgoe is 6KB 2024-02-10T05:30:11.671 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.iNsl9aBgoe 2024-02-10T05:30:11.686 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:30:11.750 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:30:11.750 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:30:11.750 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:30:11.750 INFO:teuthology.orchestra.run.smithi110.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:30:11.751 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:30:11.894 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2024-02-10T05:30:25.560 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:25.560 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-10T05:30:25.560 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:30:25.633 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:26.130 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:26.131 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-10T05:30:26.131 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-10T05:30:26.131 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-10T05:30:26.590 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:30:26.591 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-02-10T05:30:27.361 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:30:27.518 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:27.518 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:30:27.518 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-10T05:30:27.518 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-10T05:30:27.518 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-10T05:30:27.518 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-02-10T05:30:27.518 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-02-10T05:30:27.519 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-02-10T05:30:27.631 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-10T05:30:27.631 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-10T05:30:27.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-02-10T05:30:27.669 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-02-10T05:30:27.669 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-02-10T05:30:27.669 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-02-10T05:30:27.669 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-02-10T05:30:27.670 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-02-10T05:30:27.670 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-10T05:30:27.670 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-02-10T05:30:27.670 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-02-10T05:30:27.670 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-02-10T05:30:27.670 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-10T05:30:27.671 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-10T05:30:27.671 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-10T05:30:27.671 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-10T05:30:27.671 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-02-10T05:30:27.671 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-02-10T05:30:27.671 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-10T05:30:27.672 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-02-10T05:30:27.672 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-02-10T05:30:27.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:30:27.672 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-10T05:30:27.672 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-10T05:30:27.673 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-10T05:30:27.673 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-02-10T05:30:27.673 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:30:27.673 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-02-10T05:30:27.674 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-10T05:30:27.674 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-419.el9.x86_64 2024-02-10T05:30:27.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-10T05:30:28.520 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-02-10T05:30:30.081 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-10T05:30:30.106 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-02-10T05:30:30.108 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-02-10T05:30:30.123 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.EsgjQXf7Lo 2024-02-10T05:30:30.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EsgjQXf7Lo 2024-02-10T05:30:30.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.EsgjQXf7Lo 2024-02-10T05:30:30.354 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.EsgjQXf7Lo is 6KB 2024-02-10T05:30:30.402 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.EsgjQXf7Lo 2024-02-10T05:30:30.417 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-02-10T05:30:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-10T05:30:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-10T05:30:30.488 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-10T05:30:30.489 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-10T05:30:30.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-10T05:30:30.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-02-10T05:30:34.680 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-02-10T05:30:34.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-02-10T05:30:34.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:30:41.900 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-02-10T05:30:41.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-02-10T05:30:41.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:31:00.668 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-02-10T05:31:00.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T05:31:00.669 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:31:34.716 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:31:41.912 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:31:43.719 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-02-10T05:31:43.720 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:31:44.113 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-02-10T05:31:44.529 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-02-10T05:31:44.529 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:31:44.529 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-02-10T05:31:44.543 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:31:44.543 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:31:44.543 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:31:44.544 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:31:45.544 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-02-10T05:31:45.544 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:31:45.647 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-02-10T05:31:45.697 DEBUG:teuthology.parallel:result is None 2024-02-10T05:31:50.913 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-02-10T05:31:50.914 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:31:54.376 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-02-10T05:31:54.925 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-02-10T05:31:54.925 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:31:54.925 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2024-02-10T05:31:54.940 INFO:teuthology.orchestra.run.smithi110.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:31:54.940 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:31:54.940 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:31:54.941 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:31:55.942 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-02-10T05:31:55.942 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:31:56.021 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2024-02-10T05:31:56.062 DEBUG:teuthology.parallel:result is None 2024-02-10T05:32:00.672 DEBUG:teuthology.orchestra.remote:timed out 2024-02-10T05:32:09.673 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T05:32:09.674 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:32:17.340 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-02-10T05:32:17.844 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T05:32:17.844 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-419.el9.x86_64"... 2024-02-10T05:32:17.844 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-02-10T05:32:17.858 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-419.el9.x86_64 2024-02-10T05:32:17.858 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-10T05:32:17.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-10T05:32:17.858 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-10T05:32:18.860 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-02-10T05:32:18.860 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-02-10T05:32:19.010 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-02-10T05:32:19.049 DEBUG:teuthology.parallel:result is None 2024-02-10T05:32:19.049 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-10T05:32:19.058 INFO:teuthology.task.internal:Creating test directory... 2024-02-10T05:32:19.058 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:32:19.060 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:32:19.063 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-10T05:32:19.082 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-10T05:32:19.089 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-10T05:32:19.095 INFO:teuthology.task.internal:Creating archive directory... 2024-02-10T05:32:19.096 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:32:19.116 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:32:19.121 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-10T05:32:19.260 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-10T05:32:19.270 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-10T05:32:19.270 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:32:19.273 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:32:19.276 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-10T05:32:19.310 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:32:19.315 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:32:19.322 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:32:19.323 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:32:19.328 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:32:19.337 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-10T05:32:19.338 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-10T05:32:19.345 INFO:teuthology.task.internal:Configuring sudo... 2024-02-10T05:32:19.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:32:19.372 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:32:19.374 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-10T05:32:19.404 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-10T05:32:19.415 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-10T05:32:19.416 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:32:19.440 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:32:19.445 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-10T05:32:19.461 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:32:19.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:32:19.575 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:32:19.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:32:19.700 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:32:19.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:32:19.767 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:32:19.816 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:32:19.885 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:32:19.925 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:32:19.972 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:32:19.972 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:32:20.041 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:32:20.087 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-10T05:32:20.145 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:32:20.232 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-10T05:32:20.274 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:32:20.274 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-10T05:32:20.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-02-10T05:32:20.345 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-02-10T05:32:20.347 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2024-02-10T05:32:20.386 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:32:20.392 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:32:20.443 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:32:20.824 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-10T05:32:20.833 INFO:teuthology.task.internal:Starting timer... 2024-02-10T05:32:20.833 INFO:teuthology.run_tasks:Running task pcp... 2024-02-10T05:32:20.843 INFO:teuthology.run_tasks:Running task selinux... 2024-02-10T05:32:20.852 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-10T05:32:20.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-02-10T05:32:20.964 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-02-10T05:32:20.966 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-02-10T05:32:21.023 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-02-10T05:32:21.024 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2024-02-10T05:32:21.082 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2024-02-10T05:32:21.084 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-10T05:32:21.084 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-02-10T05:32:21.108 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-02-10T05:32:21.108 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-02-10T05:32:21.129 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-02-10T05:32:21.129 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2024-02-10T05:32:21.153 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2024-02-10T05:32:21.154 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive'} 2024-02-10T05:32:21.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:32:21.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:32:21.182 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:32:21.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:32:21.213 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:32:21.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:32:21.243 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-10T05:32:21.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:32:21.277 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:32:21.323 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2024-02-10T05:32:21.355 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-10T05:32:21.365 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-10T05:32:21.384 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-10T05:32:21.386 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi105.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-10T05:36:44.560 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2024-02-10T05:36:44.561 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T05:36:44.562 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:36:44.651 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-02-10T05:36:44.727 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-02-10T05:36:44.728 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-02-10T05:36:44.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:36:44.818 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-02-10T05:36:44.890 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-02-10T05:36:44.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-02-10T05:36:44.891 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-10T05:36:44.980 DEBUG:teuthology.orchestra.run.smithi110:> true 2024-02-10T05:36:45.059 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2024-02-10T05:36:45.060 INFO:teuthology.run_tasks:Running task clock... 2024-02-10T05:36:45.070 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-10T05:36:45.070 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:36:45.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:36:45.073 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:36:45.073 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:36:45.075 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-10T05:36:45.075 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:36:45.104 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:36:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:36:45.123 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:36:45.123 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:36:45.131 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-10T05:36:45.148 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-10T05:36:45.149 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-02-10T05:36:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-02-10T05:36:45.163 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-02-10T05:36:45.167 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-02-10T05:36:45.178 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2024-02-10T05:36:45.179 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:36:45.184 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:36:45.191 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2024-02-10T05:36:45.195 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:36:45.201 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:36:45.207 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-10T05:36:45.223 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-10T05:36:45.246 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:36:45.250 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:36:45.250 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-02-10T05:36:45.250 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.250 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.250 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.250 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.255 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:36:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:36:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-02-10T05:36:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.275 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:36:45.279 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:36:45.279 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-02-10T05:36:45.279 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.279 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.279 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.279 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:36:45.279 INFO:teuthology.run_tasks:Running task pexec... 2024-02-10T05:36:45.288 INFO:teuthology.task.pexec:Executing custom commands... 2024-02-10T05:36:45.289 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi029.front.sepia.ceph.com 2024-02-10T05:36:45.289 DEBUG:teuthology.orchestra.run.smithi029:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T05:36:45.289 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi105.front.sepia.ceph.com 2024-02-10T05:36:45.289 DEBUG:teuthology.orchestra.run.smithi105:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T05:36:45.290 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi110.front.sepia.ceph.com 2024-02-10T05:36:45.290 DEBUG:teuthology.orchestra.run.smithi110:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-02-10T05:36:45.891 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:59 ago on Sat 10 Feb 2024 05:35:46 AM UTC. 2024-02-10T05:36:45.910 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:49 ago on Sat 10 Feb 2024 05:35:56 AM UTC. 2024-02-10T05:36:45.918 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:59 ago on Sat 10 Feb 2024 05:35:46 AM UTC. 2024-02-10T05:36:46.045 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-02-10T05:36:46.059 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-02-10T05:36:46.059 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-02-10T05:36:46.059 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-02-10T05:36:46.059 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-02-10T05:36:46.059 INFO:teuthology.orchestra.run.smithi105.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-10T05:36:46.059 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:36:46.060 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-02-10T05:36:46.060 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-02-10T05:36:46.060 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-02-10T05:36:46.060 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:36:46.060 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.1 M 2024-02-10T05:36:46.060 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 9.8 M 2024-02-10T05:36:46.060 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-02-10T05:36:46.070 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-02-10T05:36:46.071 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2024-02-10T05:36:46.073 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2024-02-10T05:36:46.074 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2024-02-10T05:36:46.074 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:36:46.074 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.1 M 2024-02-10T05:36:46.074 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 9.8 M 2024-02-10T05:36:46.074 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout: runc x86_64 4:1.1.12-1.el9 CentOS-AppStream 3.1 M 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-02-10T05:36:46.094 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:36:46.095 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.1 M 2024-02-10T05:36:46.095 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 9.8 M 2024-02-10T05:36:46.095 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-02-10T05:36:46.266 INFO:teuthology.orchestra.run.smithi105.stdout:runc-1.1.12-1.el9.x86_64.rpm 15 MB/s | 3.1 MB 00:00 2024-02-10T05:36:46.267 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:36:46.267 INFO:teuthology.orchestra.run.smithi105.stdout:Total 15 MB/s | 3.1 MB 00:00 2024-02-10T05:36:46.267 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-02-10T05:36:46.273 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-02-10T05:36:46.273 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-02-10T05:36:46.328 INFO:teuthology.orchestra.run.smithi110.stdout:runc-1.1.12-1.el9.x86_64.rpm 12 MB/s | 3.1 MB 00:00 2024-02-10T05:36:46.329 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:36:46.329 INFO:teuthology.orchestra.run.smithi110.stdout:Total 12 MB/s | 3.1 MB 00:00 2024-02-10T05:36:46.329 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2024-02-10T05:36:46.334 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2024-02-10T05:36:46.334 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2024-02-10T05:36:46.343 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-02-10T05:36:46.343 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-02-10T05:36:46.392 INFO:teuthology.orchestra.run.smithi029.stdout:runc-1.1.12-1.el9.x86_64.rpm 11 MB/s | 3.1 MB 00:00 2024-02-10T05:36:46.393 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:36:46.393 INFO:teuthology.orchestra.run.smithi029.stdout:Total 10 MB/s | 3.1 MB 00:00 2024-02-10T05:36:46.393 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-02-10T05:36:46.399 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-02-10T05:36:46.399 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-02-10T05:36:46.404 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2024-02-10T05:36:46.404 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2024-02-10T05:36:46.473 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-02-10T05:36:46.474 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-02-10T05:36:46.550 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-02-10T05:36:46.608 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2024-02-10T05:36:46.635 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-02-10T05:36:46.714 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:46.816 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:47.142 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:47.192 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:47.499 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:47.504 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:47.504 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:36:47.504 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2024-02-10T05:36:47.504 INFO:teuthology.orchestra.run.smithi110.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-10T05:36:47.504 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:36:47.504 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2024-02-10T05:36:47.504 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:47.505 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:36:47.505 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-02-10T05:36:47.505 INFO:teuthology.orchestra.run.smithi105.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-10T05:36:47.505 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-02-10T05:36:47.505 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-02-10T05:36:47.766 DEBUG:teuthology.parallel:result is None 2024-02-10T05:36:47.788 DEBUG:teuthology.parallel:result is None 2024-02-10T05:36:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : runc-4:1.1.12-1.el9.x86_64 1/1 2024-02-10T05:36:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:36:47.815 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-02-10T05:36:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: runc-4:1.1.12-1.el9.x86_64 2024-02-10T05:36:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:36:47.815 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-02-10T05:36:48.108 DEBUG:teuthology.parallel:result is None 2024-02-10T05:36:48.108 INFO:teuthology.run_tasks:Running task nvme_loop... 2024-02-10T05:36:48.117 INFO:tasks.nvme_loop:Setting up nvme_loop on scratch devices... 2024-02-10T05:36:48.118 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:36:48.118 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:36:48.132 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:36:48.133 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-02-10T05:36:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:36:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:48.187 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-02-10T05:36:48.187 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:48.187 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:48.187 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:36:43.521235051 +0000 2024-02-10T05:36:48.188 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:21.859587998 +0000 2024-02-10T05:36:48.188 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:21.859587998 +0000 2024-02-10T05:36:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:21.859587998 +0000 2024-02-10T05:36:48.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:36:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:36:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:36:48.252 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000103759 s, 4.9 MB/s 2024-02-10T05:36:48.253 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:36:48.308 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-02-10T05:36:48.362 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:36:43.521235051 +0000 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:22.194582539 +0000 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:22.194582539 +0000 2024-02-10T05:36:48.363 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:22.194582539 +0000 2024-02-10T05:36:48.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:36:48.428 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.8942e-05 s, 5.2 MB/s 2024-02-10T05:36:48.429 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:36:48.484 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-02-10T05:36:48.538 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:36:48.538 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:48.538 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T05:36:48.538 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:48.538 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:48.538 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:36:43.521235051 +0000 2024-02-10T05:36:48.539 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:22.533577016 +0000 2024-02-10T05:36:48.539 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:22.533577016 +0000 2024-02-10T05:36:48.539 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:22.533577016 +0000 2024-02-10T05:36:48.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:36:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:36:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:36:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.7948e-05 s, 5.2 MB/s 2024-02-10T05:36:48.604 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:36:48.659 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-02-10T05:36:48.713 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:36:48.713 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:48.713 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-02-10T05:36:48.713 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:48.713 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:48.713 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:36:43.521235051 +0000 2024-02-10T05:36:48.714 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:22.866571590 +0000 2024-02-10T05:36:48.714 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:22.866571590 +0000 2024-02-10T05:36:48.714 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:22.866571590 +0000 2024-02-10T05:36:48.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:36:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:36:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:36:48.779 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 8.8815e-05 s, 5.8 MB/s 2024-02-10T05:36:48.780 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:36:48.836 DEBUG:teuthology.orchestra.run.smithi029:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-10T05:36:48.978 INFO:teuthology.orchestra.run.smithi029.stdout:loop 2024-02-10T05:36:48.979 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_1... 2024-02-10T05:36:48.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-10T05:36:49.017 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-02-10T05:36:49.054 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_11 2024-02-10T05:36:49.083 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_2... 2024-02-10T05:36:49.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-10T05:36:49.162 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-02-10T05:36:49.199 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_21 2024-02-10T05:36:49.227 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_3... 2024-02-10T05:36:49.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-10T05:36:49.304 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-02-10T05:36:49.341 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_31 2024-02-10T05:36:49.376 INFO:tasks.nvme_loop:Connecting nvme_loop smithi029:/dev/vg_nvme/lv_4... 2024-02-10T05:36:49.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-10T05:36:49.454 INFO:teuthology.orchestra.run.smithi029.stdout:1 2024-02-10T05:36:49.491 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/vg_nvme/lv_41 2024-02-10T05:36:49.518 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:36:49.518 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:36:49.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo nvme list 2024-02-10T05:36:49.638 INFO:teuthology.orchestra.run.smithi029.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-10T05:36:49.638 INFO:teuthology.orchestra.run.smithi029.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-10T05:36:49.638 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme0n1 PHMB7512009B480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-02-10T05:36:49.638 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme1n1 16be90b6ca8db55c4ed5 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:49.639 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme2n1 9486851be75fab6f415f Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:49.639 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme3n1 0a99aa7d45ac615615d4 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:49.639 INFO:teuthology.orchestra.run.smithi029.stdout:/dev/nvme4n1 e3154ab83580eaccb4d7 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:49.639 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:36:49.640 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:36:49.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/scratch_devs 2024-02-10T05:36:49.706 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:36:49.706 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:36:49.725 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:36:49.725 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-02-10T05:36:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:36:49.782 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:49.782 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-10T05:36:49.783 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:49.783 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:49.783 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:36:43.530272723 +0000 2024-02-10T05:36:49.783 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:10.724899297 +0000 2024-02-10T05:36:49.783 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:10.724899297 +0000 2024-02-10T05:36:49.783 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:10.724899297 +0000 2024-02-10T05:36:49.783 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:36:49.852 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:36:49.852 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:36:49.852 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000199963 s, 2.6 MB/s 2024-02-10T05:36:49.854 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:36:49.913 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-02-10T05:36:49.970 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:36:43.530272723 +0000 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:11.057892937 +0000 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:11.057892937 +0000 2024-02-10T05:36:49.971 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:11.057892937 +0000 2024-02-10T05:36:49.971 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:36:50.038 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:36:50.038 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:36:50.038 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000203999 s, 2.5 MB/s 2024-02-10T05:36:50.039 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:36:50.096 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-02-10T05:36:50.152 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:36:50.152 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:50.152 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-02-10T05:36:50.153 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:50.153 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:50.153 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:36:43.530272723 +0000 2024-02-10T05:36:50.153 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:11.416886080 +0000 2024-02-10T05:36:50.153 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:11.416886080 +0000 2024-02-10T05:36:50.153 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:11.416886080 +0000 2024-02-10T05:36:50.153 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:36:50.220 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:36:50.220 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:36:50.220 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000176641 s, 2.9 MB/s 2024-02-10T05:36:50.222 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:36:50.278 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-02-10T05:36:50.334 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:36:50.334 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:50.335 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-02-10T05:36:50.335 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:50.335 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:50.335 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:36:43.530272723 +0000 2024-02-10T05:36:50.335 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:11.760879510 +0000 2024-02-10T05:36:50.335 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:11.760879510 +0000 2024-02-10T05:36:50.335 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:11.760879510 +0000 2024-02-10T05:36:50.335 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:36:50.401 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:36:50.401 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:36:50.402 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000200804 s, 2.5 MB/s 2024-02-10T05:36:50.403 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:36:50.459 DEBUG:teuthology.orchestra.run.smithi105:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-10T05:36:50.606 INFO:teuthology.orchestra.run.smithi105.stdout:loop 2024-02-10T05:36:50.608 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_1... 2024-02-10T05:36:50.608 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-10T05:36:50.645 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-02-10T05:36:50.699 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_11 2024-02-10T05:36:50.737 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_2... 2024-02-10T05:36:50.738 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-10T05:36:50.816 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-02-10T05:36:50.855 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_21 2024-02-10T05:36:50.885 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_3... 2024-02-10T05:36:50.885 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-10T05:36:50.964 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-02-10T05:36:51.003 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_31 2024-02-10T05:36:51.033 INFO:tasks.nvme_loop:Connecting nvme_loop smithi105:/dev/vg_nvme/lv_4... 2024-02-10T05:36:51.033 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-10T05:36:51.112 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-02-10T05:36:51.151 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/vg_nvme/lv_41 2024-02-10T05:36:51.188 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:36:51.188 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:36:51.243 DEBUG:teuthology.orchestra.run.smithi105:> sudo nvme list 2024-02-10T05:36:51.311 INFO:teuthology.orchestra.run.smithi105.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-10T05:36:51.311 INFO:teuthology.orchestra.run.smithi105.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-10T05:36:51.311 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme0n1 PHFT620400TR400BGN INTEL SSDPEDMD400G4 1 400.09 GB / 400.09 GB 512 B + 0 B 8DV101H0 2024-02-10T05:36:51.311 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme1n1 cff449fa837008a84fc8 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:51.311 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme2n1 d2829cba1868a95de4ca Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:51.311 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme3n1 dbb18f21dd8b19ca220e Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:51.312 INFO:teuthology.orchestra.run.smithi105.stdout:/dev/nvme4n1 34583723faaf7f26cdcb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:51.313 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:36:51.313 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:36:51.313 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/scratch_devs 2024-02-10T05:36:51.379 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:36:51.379 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:36:51.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-10T05:36:51.398 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2024-02-10T05:36:51.455 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-10T05:36:51.455 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:51.456 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-10T05:36:51.456 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:51.456 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:51.456 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:36:43.555751259 +0000 2024-02-10T05:36:51.456 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:10.734269933 +0000 2024-02-10T05:36:51.456 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:10.734269933 +0000 2024-02-10T05:36:51.456 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:10.734269933 +0000 2024-02-10T05:36:51.456 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-10T05:36:51.524 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:36:51.524 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:36:51.524 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.6224e-05 s, 5.3 MB/s 2024-02-10T05:36:51.525 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-10T05:36:51.583 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2024-02-10T05:36:51.639 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:36:43.555751259 +0000 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:11.074264555 +0000 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:11.074264555 +0000 2024-02-10T05:36:51.640 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:11.074264555 +0000 2024-02-10T05:36:51.640 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-10T05:36:51.708 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:36:51.708 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:36:51.708 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.8196e-05 s, 5.8 MB/s 2024-02-10T05:36:51.709 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-10T05:36:51.767 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2024-02-10T05:36:51.823 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-10T05:36:51.823 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:51.823 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-10T05:36:51.823 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:51.823 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:51.823 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:36:43.555751259 +0000 2024-02-10T05:36:51.824 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:11.393259508 +0000 2024-02-10T05:36:51.824 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:11.393259508 +0000 2024-02-10T05:36:51.824 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:11.393259508 +0000 2024-02-10T05:36:51.824 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-10T05:36:51.890 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:36:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:36:51.891 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000120984 s, 4.2 MB/s 2024-02-10T05:36:51.891 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-10T05:36:51.948 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2024-02-10T05:36:52.003 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-10T05:36:52.003 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-10T05:36:52.003 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-02-10T05:36:52.004 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-10T05:36:52.004 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2024-02-10T05:36:52.004 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:36:43.555751259 +0000 2024-02-10T05:36:52.004 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:11.715254415 +0000 2024-02-10T05:36:52.004 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:11.715254415 +0000 2024-02-10T05:36:52.004 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:11.715254415 +0000 2024-02-10T05:36:52.004 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-10T05:36:52.069 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:36:52.069 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:36:52.069 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 8.1178e-05 s, 6.3 MB/s 2024-02-10T05:36:52.071 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-10T05:36:52.126 DEBUG:teuthology.orchestra.run.smithi110:> grep '^nvme_loop' /proc/modules || sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype 2024-02-10T05:36:52.270 INFO:teuthology.orchestra.run.smithi110.stdout:loop 2024-02-10T05:36:52.271 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_1... 2024-02-10T05:36:52.272 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn 2024-02-10T05:36:52.309 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-02-10T05:36:52.346 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_11 2024-02-10T05:36:52.375 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_2... 2024-02-10T05:36:52.375 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1 && echo -n /dev/vg_nvme/lv_2 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_2/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_2 /sys/kernel/config/nvmet/ports/1/subsystems/lv_2 && sudo nvme connect -t loop -n lv_2 -q hostnqn 2024-02-10T05:36:52.453 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-02-10T05:36:52.493 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_21 2024-02-10T05:36:52.535 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_3... 2024-02-10T05:36:52.535 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1 && echo -n /dev/vg_nvme/lv_3 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_3/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_3 /sys/kernel/config/nvmet/ports/1/subsystems/lv_3 && sudo nvme connect -t loop -n lv_3 -q hostnqn 2024-02-10T05:36:52.615 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-02-10T05:36:52.653 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_31 2024-02-10T05:36:52.689 INFO:tasks.nvme_loop:Connecting nvme_loop smithi110:/dev/vg_nvme/lv_4... 2024-02-10T05:36:52.689 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1 && echo -n /dev/vg_nvme/lv_4 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_4/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_4 /sys/kernel/config/nvmet/ports/1/subsystems/lv_4 && sudo nvme connect -t loop -n lv_4 -q hostnqn 2024-02-10T05:36:52.771 INFO:teuthology.orchestra.run.smithi110.stdout:1 2024-02-10T05:36:52.810 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/vg_nvme/lv_41 2024-02-10T05:36:52.840 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:36:52.840 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:36:52.901 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme list 2024-02-10T05:36:52.968 INFO:teuthology.orchestra.run.smithi110.stdout:Node SN Model Namespace Usage Format FW Rev 2024-02-10T05:36:52.968 INFO:teuthology.orchestra.run.smithi110.stdout:---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- 2024-02-10T05:36:52.968 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme0n1 PHMB7476002T480DGN INTEL SSDPED1D480GA 1 480.10 GB / 480.10 GB 512 B + 0 B E2010325 2024-02-10T05:36:52.968 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme1n1 761888bf8420e14a1f25 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:52.968 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme2n1 e1aea55af5723b83a713 Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:52.968 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme3n1 ac739ecbfb90b66518ec Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:52.968 INFO:teuthology.orchestra.run.smithi110.stdout:/dev/nvme4n1 633089ac8c83c16ab9fb Linux 1 96.00 GB / 96.00 GB 512 B + 0 B 5.14.0-4 2024-02-10T05:36:52.969 INFO:tasks.nvme_loop:new_devs ['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:36:52.969 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:36:52.969 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/scratch_devs 2024-02-10T05:36:53.037 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-10T05:36:53.130 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'global': {'mon election default strategy': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-10T05:36:53.130 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:36:53.131 INFO:tasks.cephadm:Cluster fsid is 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:36:53.131 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-10T05:36:53.131 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.b': '172.21.15.105', 'mon.c': '172.21.15.110'} 2024-02-10T05:36:53.131 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-02-10T05:36:53.131 INFO:tasks.cephadm:First mgr is y 2024-02-10T05:36:53.131 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-10T05:36:53.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-02-10T05:36:53.162 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-02-10T05:36:53.194 DEBUG:teuthology.orchestra.run.smithi110:> sudo hostname $(hostname -s) 2024-02-10T05:36:53.221 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-10T05:36:53.222 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:36:53.380 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76638/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1361-g290212a1', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1361.g290212a1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-09 18:34:21.832169', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-10T05:36:53.545 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref main, sha1 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:36:53.547 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm 2024-02-10T05:36:53.547 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:54.297 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 05:36 /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:54.298 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:55.086 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 05:36 /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:55.086 DEBUG:teuthology.orchestra.run.smithi110:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:55.850 INFO:teuthology.orchestra.run.smithi110.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 10 05:36 /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:55.850 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:55.866 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:55.885 DEBUG:teuthology.orchestra.run.smithi110:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-10T05:36:55.916 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on all hosts... 2024-02-10T05:36:55.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T05:36:55.919 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T05:36:55.927 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-10T05:36:56.755 INFO:teuthology.orchestra.run.smithi110.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:36:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:36:57.162 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:37:25.203 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-02-10T05:37:25.203 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T05:37:25.203 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T05:37:25.203 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-02-10T05:37:25.203 INFO:teuthology.orchestra.run.smithi105.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T05:37:25.203 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-02-10T05:37:25.204 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-02-10T05:39:12.595 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-02-10T05:39:12.596 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T05:39:12.596 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T05:39:12.596 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-02-10T05:39:12.596 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T05:39:12.596 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-02-10T05:39:12.596 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-02-10T05:39:14.048 INFO:teuthology.orchestra.run.smithi110.stdout:{ 2024-02-10T05:39:14.048 INFO:teuthology.orchestra.run.smithi110.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-10T05:39:14.048 INFO:teuthology.orchestra.run.smithi110.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-10T05:39:14.048 INFO:teuthology.orchestra.run.smithi110.stdout: "repo_digests": [ 2024-02-10T05:39:14.049 INFO:teuthology.orchestra.run.smithi110.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-10T05:39:14.049 INFO:teuthology.orchestra.run.smithi110.stdout: ] 2024-02-10T05:39:14.049 INFO:teuthology.orchestra.run.smithi110.stdout:} 2024-02-10T05:39:14.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-02-10T05:39:14.107 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-02-10T05:39:14.144 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph 2024-02-10T05:39:14.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-02-10T05:39:14.202 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-02-10T05:39:14.233 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /etc/ceph 2024-02-10T05:39:14.259 INFO:tasks.cephadm:Writing seed config... 2024-02-10T05:39:14.260 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-10T05:39:14.260 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-10T05:39:14.260 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-02-10T05:39:14.261 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-02-10T05:39:14.261 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-10T05:39:14.261 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-10T05:39:14.261 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-10T05:39:14.261 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-10T05:39:14.261 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-10T05:39:14.261 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-10T05:39:14.262 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:39:14.262 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-10T05:39:14.278 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 65498394-c7d6-11ee-95b8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-02-10T05:39:14.279 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a.service 2024-02-10T05:39:14.321 DEBUG:teuthology.orchestra.run.smithi029:mgr.y> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.y.service 2024-02-10T05:39:14.363 INFO:tasks.cephadm:Bootstrapping... 2024-02-10T05:39:14.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -v bootstrap --fsid 65498394-c7d6-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:39:14.562 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-02-10T05:39:14.563 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', '-v', 'bootstrap', '--fsid', '65498394-c7d6-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-02-10T05:39:14.587 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:39:14.587 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-10T05:39:14.588 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-02-10T05:39:14.610 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:39:14.611 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-02-10T05:39:14.611 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-02-10T05:39:14.617 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T05:39:14.618 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:39:14.623 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T05:39:14.623 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-02-10T05:39:14.629 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-02-10T05:39:14.635 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-02-10T05:39:14.635 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-02-10T05:39:14.635 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-02-10T05:39:14.658 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-10T05:39:14.658 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-10T05:39:14.659 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-02-10T05:39:14.659 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-02-10T05:39:14.665 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-10T05:39:14.665 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-10T05:39:14.670 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-10T05:39:14.671 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-02-10T05:39:14.677 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-02-10T05:39:14.683 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-02-10T05:39:14.683 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-02-10T05:39:14.683 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-02-10T05:39:14.683 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:14.683 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 140172952099424 on /run/cephadm/65498394-c7d6-11ee-95b8-87774f69a715.lock 2024-02-10T05:39:14.683 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 140172952099424 acquired on /run/cephadm/65498394-c7d6-11ee-95b8-87774f69a715.lock 2024-02-10T05:39:14.684 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-02-10T05:39:14.684 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-02-10T05:39:14.684 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-02-10T05:39:14.687 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-02-10T05:39:14.687 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-02-10T05:39:14.689 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-10T05:39:14.689 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-10T05:39:14.689 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-10T05:39:14.692 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-10T05:39:14.692 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-10T05:39:14.692 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T05:39:14.692 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2024-02-10T05:39:14.692 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-02-10T05:39:14.692 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-10T05:39:14.693 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-02-10T05:39:14.693 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-02-10T05:39:14.693 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-10T05:39:14.693 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-10T05:39:14.694 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:39:15.257 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:39:15.257 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-10T05:39:15.258 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-02-10T05:39:15.258 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:5fe0e97a267976e17c7dcb240cf9455472de6876101ae78af4bdf182158b6d8d 2024-02-10T05:39:15.258 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-10T05:39:15.258 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:39:15.258 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-10T05:39:16.202 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T05:39:16.202 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-10T05:39:16.202 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-02-10T05:39:17.124 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-02-10T05:39:17.125 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-02-10T05:39:18.152 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAFDMdl0C8uHBAA4Q3/yFUbHRZlT0pC6uGU9w== 2024-02-10T05:39:19.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQAGDMdlGSPNIBAAv24dR2QTlMqkvGmE6ry7nA== 2024-02-10T05:39:33.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQALDMdlGLdvBRAAUNA4gZUUd+DbAkIasDtXNg== 2024-02-10T05:39:33.866 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-02-10T05:39:35.192 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:39:35.192 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-10T05:39:35.192 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:35.192 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:39:35.192 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-10T05:39:35.193 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-02-10T05:39:35.193 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:35.193 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-10T05:39:35.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:39:35.193 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-02-10T05:39:36.379 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.a on 2024-02-10T05:39:36.739 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-10T05:39:36.912 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-65498394-c7d6-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-65498394-c7d6-11ee-95b8-87774f69a715.target. 2024-02-10T05:39:36.912 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-65498394-c7d6-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-65498394-c7d6-11ee-95b8-87774f69a715.target. 2024-02-10T05:39:37.151 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a 2024-02-10T05:39:37.152 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-10T05:39:37.311 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-65498394-c7d6-11ee-95b8-87774f69a715.target.wants/ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-65498394-c7d6-11ee-95b8-87774f69a715@.service. 2024-02-10T05:39:37.792 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:39:37.792 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-02-10T05:39:37.798 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:39:37.798 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-02-10T05:39:37.798 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-02-10T05:39:37.798 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T05:39:37.799 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-02-10T05:39:37.799 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-02-10T05:39:38.876 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:38 smithi029 ceph-mon[25200]: from='client.? 172.21.15.29:0/4165939437' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.549125s) 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-10T05:39:38.880 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-02-10T05:39:38.881 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-02-10T05:39:40.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:39 smithi029 ceph-mon[25200]: from='client.? 172.21.15.29:0/649210849' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:39:40.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:39 smithi029 ceph-mon[25200]: from='client.? 172.21.15.29:0/649210849' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T05:39:42.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = False 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:42.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T05:39:42.106 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T05:39:42.106 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T05:39:42.106 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-02-10T05:39:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:42 smithi029 ceph-mon[25200]: from='client.? 172.21.15.29:0/2073408972' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:39:43.192 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-02-10T05:39:43.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:43 smithi029 systemd[1]: Stopping Ceph mon.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:39:43.794 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:43 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a[25177]: 2024-02-10T05:39:43.478+0000 7f2e434e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:39:43.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:43 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a[25177]: 2024-02-10T05:39:43.478+0000 7f2e434e1700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-10T05:39:43.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:43 smithi029 podman[25533]: 2024-02-10 05:39:43.55034877 +0000 UTC m=+0.189368806 container died 75d837aebe2a2d1f860c93097139766981fc38c7cc350e4e241a214926768f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, RELEASE=main-290212a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-02-10T05:39:44.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:43 smithi029 podman[25533]: 2024-02-10 05:39:43.885780899 +0000 UTC m=+0.524800939 container cleanup 75d837aebe2a2d1f860c93097139766981fc38c7cc350e4e241a214926768f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_BRANCH=HEAD) 2024-02-10T05:39:44.062 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:43 smithi029 bash[25533]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a 2024-02-10T05:39:44.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 podman[25552]: 2024-02-10 05:39:44.06245522 +0000 UTC m=+0.507769597 container remove 75d837aebe2a2d1f860c93097139766981fc38c7cc350e4e241a214926768f58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-02-10T05:39:44.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-10T05:39:44.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 systemd[1]: Stopped Ceph mon.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:39:44.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 systemd[1]: Starting Ceph mon.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:39:44.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 podman[25654]: 2024-02-10 05:39:44.436453288 +0000 UTC m=+0.019198417 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:39:44.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 podman[25654]: 2024-02-10 05:39:44.539316473 +0000 UTC m=+0.122061629 container create 2013fc10e01e409a0041c30ae853b6ad24da5e6c2170216e7c86ad282d4d128f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T05:39:44.815 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-10T05:39:45.022 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 podman[25654]: 2024-02-10 05:39:44.728005941 +0000 UTC m=+0.310751058 container init 2013fc10e01e409a0041c30ae853b6ad24da5e6c2170216e7c86ad282d4d128f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-10T05:39:45.022 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 podman[25654]: 2024-02-10 05:39:44.733029274 +0000 UTC m=+0.315774391 container start 2013fc10e01e409a0041c30ae853b6ad24da5e6c2170216e7c86ad282d4d128f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-10T05:39:45.022 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:39:45.022 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 7 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: pidfile_write: ignore empty --pid-file 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: load: jerasure load: lrc 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Git sha 0 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: DB SUMMARY 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: DB Session ID: KA9723I0L72TH6OT1A0A 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 85120 ; 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:39:45.023 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.env: 0x5601b0d29ae0 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.info_log: 0x5601b1ae58c0 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:39:45.024 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.statistics: (nil) 2024-02-10T05:39:45.025 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.use_fsync: 0 2024-02-10T05:39:45.025 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:39:45.025 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:39:45.025 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:39:45.025 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.db_log_dir: 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.wal_dir: 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:39:45.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.write_buffer_manager: 0x5601b1ab9400 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:39:45.027 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.unordered_write: 0 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.row_cache: None 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.wal_filter: None 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:39:45.028 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.wal_compression: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:39:45.029 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_open_files: -1 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:39:45.030 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Compression algorithms supported: 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kZSTD supported: 0 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:39:45.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.merge_operator: 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_filter: None 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601b1ae59c0) 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:39:45.032 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x5601b1ae9090 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-02-10T05:39:45.033 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-02-10T05:39:45.034 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression: NoCompression 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:39:45.035 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.num_levels: 7 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:39:45.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:39:45.037 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:39:45.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:39:45.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:39:45.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:39:45.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:39:45.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:39:45.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:39:45.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.ttl: 2592000 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:39:45.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: de93ac14-d7a9-4e41-9187-850e29ac9635 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543584761893, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543584762720, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81501, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 261, "table_properties": {"data_size": 79688, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10043, "raw_average_key_size": 47, "raw_value_size": 73984, "raw_average_value_size": 348, "num_data_blocks": 10, "num_entries": 212, "num_filter_entries": 212, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707543584, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "de93ac14-d7a9-4e41-9187-850e29ac9635", "db_session_id": "KA9723I0L72TH6OT1A0A", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543584762804, "job": 1, "event": "recovery_finished"} 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-10T05:39:45.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5601b1bda000 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: DB pointer 0x5601b1bc4000 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-02-10T05:39:45.043 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 2/0 81.43 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 81.43 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-02-10T05:39:45.044 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 27.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 27.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x5601b1ae9090#7 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:45.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???) e1 preinit fsid 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???).mds e1 new map 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???).mds e1 print_map 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 bash[25654]: 2013fc10e01e409a0041c30ae853b6ad24da5e6c2170216e7c86ad282d4d128f 2024-02-10T05:39:45.046 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 systemd[1]: Started Ceph mon.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:39:45.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-10T05:39:45.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: monmap e1: 1 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:39:45.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: fsmap 2024-02-10T05:39:45.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: osdmap e1: 0 total, 0 up, 0 in 2024-02-10T05:39:45.047 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:44 smithi029 ceph-mon[25692]: mgrmap e1: no daemons active 2024-02-10T05:39:45.877 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-10T05:39:46.429 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:39:46.429 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-02-10T05:39:46.429 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-10T05:39:46.430 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-10T05:39:46.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:46 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/4010588169' entity='client.admin' 2024-02-10T05:39:46.790 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.y 2024-02-10T05:39:46.790 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.y.service: Unit ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.y.service not loaded. 2024-02-10T05:39:46.952 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-65498394-c7d6-11ee-95b8-87774f69a715.target.wants/ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-65498394-c7d6-11ee-95b8-87774f69a715@.service. 2024-02-10T05:39:47.287 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:47 smithi029 podman[25937]: 2024-02-10 05:39:47.085941492 +0000 UTC m=+0.017526800 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-10T05:39:47.522 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:39:47.522 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-02-10T05:39:47.529 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:39:47.529 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-02-10T05:39:47.529 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-02-10T05:39:47.529 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-02-10T05:39:47.537 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:39:47.537 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-02-10T05:39:47.544 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:39:47.544 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-02-10T05:39:47.544 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-02-10T05:39:47.544 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-10T05:39:47.544 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-02-10T05:39:47.545 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-02-10T05:39:47.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:47 smithi029 podman[25937]: 2024-02-10 05:39:47.375793412 +0000 UTC m=+0.307378706 container init 9698232a4e27b6cac4881ae53b54fee31bc193ea3c7ad30c5de87f21ddd30779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T05:39:47.635 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:47 smithi029 podman[25937]: 2024-02-10 05:39:47.381073025 +0000 UTC m=+0.312658320 container start 9698232a4e27b6cac4881ae53b54fee31bc193ea3c7ad30c5de87f21ddd30779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:39:47.635 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:47 smithi029 bash[25937]: 9698232a4e27b6cac4881ae53b54fee31bc193ea3c7ad30c5de87f21ddd30779 2024-02-10T05:39:47.635 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:47 smithi029 systemd[1]: Started Ceph mgr.y for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:39:48.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:48 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3550980897' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:39:48.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:47 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:47.950+0000 7f9952353200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:39:48.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:48 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:48.049+0000 7f9952353200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "65498394-c7d6-11ee-95b8-87774f69a715", 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:39:48.696 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:48.697 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:39:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:39:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:39:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:39:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:39:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:39:48.698 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:39:48.701 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:39:48.702 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:48.703 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:39:37.776276+0000", 2024-02-10T05:39:48.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:48.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:48.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:39:48.704 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:39:48.704 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-02-10T05:39:48.961 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:48 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:48.723+0000 7f9952353200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:39:48.961 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:48 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:48.817+0000 7f9952353200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:39:49.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:48 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:48.960+0000 7f9952353200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:39:49.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:49 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:49.058+0000 7f9952353200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:39:49.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:49 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:49.142+0000 7f9952353200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:39:49.704 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:49 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:49.409+0000 7f9952353200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:39:49.705 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:49 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:49.520+0000 7f9952353200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:39:49.705 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:49 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:49.607+0000 7f9952353200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:39:49.705 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:49 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:49.704+0000 7f9952353200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:39:50.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:49 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:49.881+0000 7f9952353200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:39:51.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:50 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:50.887+0000 7f9952353200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:39:51.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:51 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:51.026+0000 7f9952353200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:39:51.467 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:51 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3685361534' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:39:51.467 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:51 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:51.374+0000 7f9952353200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:39:51.467 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:51 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:51.466+0000 7f9952353200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:39:51.829 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:51 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:51.688+0000 7f9952353200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:39:51.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:51.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:39:51.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "65498394-c7d6-11ee-95b8-87774f69a715", 2024-02-10T05:39:51.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:39:51.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:39:51.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:39:51.834 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:39:51.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:39:51.836 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:39:51.839 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:39:51.840 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:51.841 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:39:37.776276+0000", 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:39:51.842 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (2/15)... 2024-02-10T05:39:52.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:51 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:51.877+0000 7f9952353200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:39:52.426 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:52 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:52.165+0000 7f9952353200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:39:52.426 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:52 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:52.425+0000 7f9952353200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:39:52.884 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:52 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:52.665+0000 7f9952353200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:39:53.884 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:53 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:53.487+0000 7f9952353200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:39:54.294 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:54 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:54.080+0000 7f9952353200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:39:54.294 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:54 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:54.181+0000 7f9952353200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:39:54.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:54 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3159093181' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:39:54.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:39:54 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:39:54.410+0000 7f9952353200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:39:54.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:54.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:39:54.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "65498394-c7d6-11ee-95b8-87774f69a715", 2024-02-10T05:39:54.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:39:54.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:39:54.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:39:54.915 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:39:54.916 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:39:54.917 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:54.921 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:39:54.922 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:54.923 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:39:37.776276+0000", 2024-02-10T05:39:54.923 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:54.923 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:54.923 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:39:54.923 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:39:54.923 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (3/15)... 2024-02-10T05:39:55.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: Activating manager daemon y 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: mgrmap e2: y(active, starting, since 0.00297645s) 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: Manager daemon y is now available 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' 2024-02-10T05:39:55.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' 2024-02-10T05:39:55.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:55 smithi029 ceph-mon[25692]: from='mgr.14100 172.21.15.29:0/770343065' entity='mgr.y' 2024-02-10T05:39:56.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:56 smithi029 ceph-mon[25692]: mgrmap e3: y(active, since 1.00696s) 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "65498394-c7d6-11ee-95b8-87774f69a715", 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:58.278 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-10T05:39:58.279 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-10T05:39:58.280 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-10T05:39:58.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:58.283 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-10T05:39:58.284 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-10T05:39:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-02-10T05:39:37.776276+0000", 2024-02-10T05:39:58.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-10T05:39:58.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-02-10T05:39:58.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-10T05:39:58.286 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:39:58.286 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-02-10T05:39:58.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:58 smithi029 ceph-mon[25692]: mgrmap e4: y(active, since 3s) 2024-02-10T05:39:58.613 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:58 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2902363615' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-10T05:39:59.529 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:59 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/222881390' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-10T05:39:59.530 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:39:59 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/222881390' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-10T05:39:59.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:59.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-02-10T05:39:59.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:39:59.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-10T05:39:59.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-10T05:39:59.533 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-10T05:39:59.534 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-02-10T05:40:01.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:00 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/1426495600' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-10T05:40:01.083 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:00 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ignoring --setuser ceph since I am not root 2024-02-10T05:40:01.083 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:00 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ignoring --setgroup ceph since I am not root 2024-02-10T05:40:01.603 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:01 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:01.346+0000 7f9cf2c51200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:40:01.603 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:01 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:01.445+0000 7f9cf2c51200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:40:01.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:01 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/1426495600' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-10T05:40:01.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:01 smithi029 ceph-mon[25692]: mgrmap e5: y(active, since 6s) 2024-02-10T05:40:02.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:02.116+0000 7f9cf2c51200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:40:02.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:02.217+0000 7f9cf2c51200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:40:02.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:02.360+0000 7f9cf2c51200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:40:02.739 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:02.460+0000 7f9cf2c51200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:40:02.739 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:02.545+0000 7f9cf2c51200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-02-10T05:40:02.742 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-02-10T05:40:03.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3814436188' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:40:03.033 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:02.824+0000 7f9cf2c51200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:40:03.033 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:02.937+0000 7f9cf2c51200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:40:03.033 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:03 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:03.031+0000 7f9cf2c51200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:40:03.308 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:03 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:03.129+0000 7f9cf2c51200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:40:03.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:03 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:03.308+0000 7f9cf2c51200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:40:04.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:04 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:04.293+0000 7f9cf2c51200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:40:04.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:04 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:04.427+0000 7f9cf2c51200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:40:05.060 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:04 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:04.770+0000 7f9cf2c51200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:40:05.060 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:04 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:04.862+0000 7f9cf2c51200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:40:05.061 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:05 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:05.060+0000 7f9cf2c51200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:40:05.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:05 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:05.247+0000 7f9cf2c51200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:40:05.816 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:05 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:05.537+0000 7f9cf2c51200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:40:06.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:05 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:05.816+0000 7f9cf2c51200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:40:06.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:06 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:06.056+0000 7f9cf2c51200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:40:07.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:06 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:06.878+0000 7f9cf2c51200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:40:07.782 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:07.460+0000 7f9cf2c51200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:40:07.782 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:07.560+0000 7f9cf2c51200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:40:07.782 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:07.781+0000 7f9cf2c51200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:40:08.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: Active manager daemon y restarted 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: Activating manager daemon y 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: osdmap e2: 0 total, 0 up, 0 in 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: mgrmap e6: y(active, starting, since 0.00352376s) 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:40:08.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: Manager daemon y is now available 2024-02-10T05:40:08.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:08.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:08.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:08.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:07 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:09.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:08 smithi029 ceph-mon[25692]: Found migration_current of "None". Setting to last migration. 2024-02-10T05:40:09.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:08 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:09.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:08 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:40:09.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:08 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:40:09.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:08 smithi029 ceph-mon[25692]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:40:09.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:08 smithi029 ceph-mon[25692]: mgrmap e7: y(active, since 1.00535s) 2024-02-10T05:40:09.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:08 smithi029 ceph-mon[25692]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:40:09.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:40:09.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-10T05:40:09.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T05:40:09.382 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:40:09.382 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-02-10T05:40:09.382 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-02-10T05:40:11.007 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:10 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:11.007 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:10 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:11.007 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:10 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:11.008 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:10 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:11.835 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-10T05:40:11.835 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-02-10T05:40:11.950 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:11 smithi029 ceph-mon[25692]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:11.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:11 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:10] ENGINE Bus STARTING 2024-02-10T05:40:11.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:11 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:10] ENGINE Serving on http://172.21.15.29:8765 2024-02-10T05:40:11.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:11 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:10] ENGINE Serving on https://172.21.15.29:7150 2024-02-10T05:40:11.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:11 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:10] ENGINE Bus STARTED 2024-02-10T05:40:11.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:11 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:11.951 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:11 smithi029 ceph-mon[25692]: mgrmap e8: y(active, since 2s) 2024-02-10T05:40:12.952 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-mon[25692]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:12.952 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:12.952 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:12.952 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: Generating public/private rsa key pair. 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: Your identification has been saved in /tmp/tmp5to91c9b/key. 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: Your public key has been saved in /tmp/tmp5to91c9b/key.pub. 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: The key fingerprint is: 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: SHA256:aFaUmHMQJnyfTAHrFpWGnL50XmwkkiiSyI4DcbCZFnM ceph-65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: The key's randomart image is: 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: +---[RSA 3072]----+ 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: |=*.E.o=X++ | 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: |=*= oo@+B . | 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: |Bo . ooO.= | 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: |+. .o+= + | 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: | . .*+So | 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: | +. . | 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: | | 2024-02-10T05:40:12.953 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: | | 2024-02-10T05:40:12.954 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: | | 2024-02-10T05:40:12.954 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: +----[SHA256]-----+ 2024-02-10T05:40:13.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:13 smithi029 ceph-mon[25692]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:13.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:13 smithi029 ceph-mon[25692]: Generating ssh key... 2024-02-10T05:40:14.494 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoL1KgqdJMuGGLulpOEU7LwfXPy53zHBCgCzcvFID5VFnGDqUi8cN3YBuAaqQqd9NeYAttZxP/GK6gXC+qlvQtRe5fAMKIDUjdipZEuLYLD+XZXyCnSyppbl2zbF5xIeRybMDroySIYYF7rOBmz6QXfKpnq3+hsdLI3x+7FeFZnomqgQeKFbii2AgVDgvggBmNRrqrn8wULXH0c7ODXSHLLx1IovJrsos1+hQwRvh32DjyHbgYn15PJlUSPRgfvs71A3ydwD5iJ1UUXLm8AirwRC34V8NRQpX3Hhkg2bB8HZGQjdGNDzNTbehXvdtaYwsJrQ5JyMg3BIfvG3SCSabhwml72YhGOjpJxjvvDwtXO4saHKjen3eDk+a2uU4p4aLy6kd2vkB+fIsQTSqQPZO89aWzrLUSqN+KqH0Ztw6gMynHZRGkL5L9i24DWbAU6fE18h1C2Fk8SWXALCy5B5bbu8PCtrfs17xnYKFgGjhGY9Tcap4TbLJFQodUdUIIG0= ceph-65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:40:14.494 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:40:14.494 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-02-10T05:40:14.495 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-02-10T05:40:15.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:15 smithi029 ceph-mon[25692]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:17.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:16 smithi029 ceph-mon[25692]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:18.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:17 smithi029 ceph-mon[25692]: Deploying cephadm binary to smithi029 2024-02-10T05:40:20.171 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-02-10T05:40:20.171 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mon service... 2024-02-10T05:40:20.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:20 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:20.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:20 smithi029 ceph-mon[25692]: Added host smithi029 2024-02-10T05:40:20.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:20 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:21.754 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-10T05:40:21.754 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mgr service... 2024-02-10T05:40:22.003 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:21 smithi029 ceph-mon[25692]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:22.003 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:21 smithi029 ceph-mon[25692]: Saving service mon spec with placement count:5 2024-02-10T05:40:22.004 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:21 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:23.190 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-10T05:40:23.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:23 smithi029 ceph-mon[25692]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:23.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:23 smithi029 ceph-mon[25692]: Saving service mgr spec with placement count:2 2024-02-10T05:40:23.777 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:23 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:25.003 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:24 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/428718775' entity='client.admin' 2024-02-10T05:40:25.003 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:24 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:26.080 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-02-10T05:40:26.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:26 smithi029 ceph-mon[25692]: from='mgr.14120 172.21.15.29:0/2308544190' entity='mgr.y' 2024-02-10T05:40:26.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:26 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3394181429' entity='client.admin' 2024-02-10T05:40:27.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:27 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3721552542' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-10T05:40:27.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:27 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ignoring --setuser ceph since I am not root 2024-02-10T05:40:27.385 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:27 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ignoring --setgroup ceph since I am not root 2024-02-10T05:40:27.763 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:27 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:27.670+0000 7fa2a0e5a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:40:28.047 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:27 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:27.766+0000 7fa2a0e5a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:40:28.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:28 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3721552542' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-10T05:40:28.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:28 smithi029 ceph-mon[25692]: mgrmap e9: y(active, since 19s) 2024-02-10T05:40:28.674 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:28.431+0000 7fa2a0e5a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:40:28.674 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:28.528+0000 7fa2a0e5a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:40:28.674 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:28.673+0000 7fa2a0e5a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:40:29.081 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:28.771+0000 7fa2a0e5a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:40:29.081 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:28.861+0000 7fa2a0e5a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:40:29.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:29 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3523508727' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-10T05:40:29.336 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:29 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:29.133+0000 7fa2a0e5a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:40:29.336 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:29 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:29.245+0000 7fa2a0e5a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:40:29.336 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:29 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:29.335+0000 7fa2a0e5a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:40:29.391 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:40:29.391 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-10T05:40:29.391 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-10T05:40:29.392 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-02-10T05:40:29.392 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-10T05:40:29.392 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:40:29.392 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-02-10T05:40:29.392 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-02-10T05:40:29.618 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:29 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:29.439+0000 7fa2a0e5a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:40:29.618 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:29 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:29.618+0000 7fa2a0e5a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:40:30.884 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:30 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:30.615+0000 7fa2a0e5a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:40:30.884 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:30 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:30.749+0000 7fa2a0e5a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:40:31.379 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:31 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:31.086+0000 7fa2a0e5a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:40:31.380 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:31 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:31.179+0000 7fa2a0e5a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:40:31.380 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:31 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:31.379+0000 7fa2a0e5a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:40:31.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:31 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:31.568+0000 7fa2a0e5a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:40:32.122 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:31 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:31.861+0000 7fa2a0e5a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:40:32.122 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:32 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:32.121+0000 7fa2a0e5a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:40:32.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:32 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:32.362+0000 7fa2a0e5a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:40:33.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:33 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:33.187+0000 7fa2a0e5a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:40:34.127 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:33 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:33.801+0000 7fa2a0e5a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:40:34.127 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:33 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:33.904+0000 7fa2a0e5a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:40:34.127 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:40:34.127+0000 7fa2a0e5a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:40:34.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: Active manager daemon y restarted 2024-02-10T05:40:34.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: Activating manager daemon y 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: osdmap e3: 0 total, 0 up, 0 in 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: mgrmap e10: y(active, starting, since 0.00319693s) 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: Manager daemon y is now available 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:34.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:34 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:35.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:40:35.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:40:35.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:35 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:34] ENGINE Bus STARTING 2024-02-10T05:40:35.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:35 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:34] ENGINE Serving on https://172.21.15.29:7150 2024-02-10T05:40:35.511 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:35 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:35] ENGINE Serving on http://172.21.15.29:8765 2024-02-10T05:40:35.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:35 smithi029 ceph-mon[25692]: [10/Feb/2024:05:40:35] ENGINE Bus STARTED 2024-02-10T05:40:35.512 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:35 smithi029 ceph-mon[25692]: mgrmap e11: y(active, since 1.00545s) 2024-02-10T05:40:35.765 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-02-10T05:40:35.765 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-10T05:40:35.766 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-10T05:40:35.766 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-02-10T05:40:35.766 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-02-10T05:40:35.766 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-02-10T05:40:36.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:36 smithi029 ceph-mon[25692]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-10T05:40:36.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:36 smithi029 ceph-mon[25692]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-10T05:40:37.455 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:37 smithi029 ceph-mon[25692]: mgrmap e12: y(active, since 2s) 2024-02-10T05:40:37.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:37 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:37.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:37 smithi029 ceph-mon[25692]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:37.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:37 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:37.456 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:37 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:37.458 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-10T05:40:37.459 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-02-10T05:40:38.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:38.795 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:39.491 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$ohWhsYyLUb1mStabPBsoMOG5mEmke/Qe8RwRLuXa7bPq66iWP/bZW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707543638, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-10T05:40:39.491 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-02-10T05:40:39.760 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:39 smithi029 ceph-mon[25692]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:40.878 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:40 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3611387412' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-10T05:40:41.150 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-02-10T05:40:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-10T05:40:41.160 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-02-10T05:40:41.166 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-10T05:40:41.167 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-02-10T05:40:41.167 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-02-10T05:40:41.167 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-10T05:40:41.169 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-02-10T05:40:41.169 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:41.169 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-02-10T05:40:41.169 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-02-10T05:40:41.169 INFO:teuthology.orchestra.run.smithi029.stdout: Password: 554u1fn53i 2024-02-10T05:40:41.169 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:41.170 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config directory 2024-02-10T05:40:41.170 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling autotune for osd_memory_target 2024-02-10T05:40:42.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:41 smithi029 ceph-mon[25692]: mgrmap e13: y(active, since 6s) 2024-02-10T05:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-10T05:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-10T05:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-02-10T05:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-02-10T05:40:43.901 INFO:tasks.cephadm:Fetching config... 2024-02-10T05:40:43.902 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:40:43.902 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-10T05:40:43.917 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-10T05:40:43.917 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:40:43.917 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-10T05:40:43.971 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-10T05:40:43.971 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:40:43.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-10T05:40:44.039 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-10T05:40:44.039 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:40:44.039 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-10T05:40:44.093 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-10T05:40:44.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoL1KgqdJMuGGLulpOEU7LwfXPy53zHBCgCzcvFID5VFnGDqUi8cN3YBuAaqQqd9NeYAttZxP/GK6gXC+qlvQtRe5fAMKIDUjdipZEuLYLD+XZXyCnSyppbl2zbF5xIeRybMDroySIYYF7rOBmz6QXfKpnq3+hsdLI3x+7FeFZnomqgQeKFbii2AgVDgvggBmNRrqrn8wULXH0c7ODXSHLLx1IovJrsos1+hQwRvh32DjyHbgYn15PJlUSPRgfvs71A3ydwD5iJ1UUXLm8AirwRC34V8NRQpX3Hhkg2bB8HZGQjdGNDzNTbehXvdtaYwsJrQ5JyMg3BIfvG3SCSabhwml72YhGOjpJxjvvDwtXO4saHKjen3eDk+a2uU4p4aLy6kd2vkB+fIsQTSqQPZO89aWzrLUSqN+KqH0Ztw6gMynHZRGkL5L9i24DWbAU6fE18h1C2Fk8SWXALCy5B5bbu8PCtrfs17xnYKFgGjhGY9Tcap4TbLJFQodUdUIIG0= ceph-65498394-c7d6-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:40:44.172 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoL1KgqdJMuGGLulpOEU7LwfXPy53zHBCgCzcvFID5VFnGDqUi8cN3YBuAaqQqd9NeYAttZxP/GK6gXC+qlvQtRe5fAMKIDUjdipZEuLYLD+XZXyCnSyppbl2zbF5xIeRybMDroySIYYF7rOBmz6QXfKpnq3+hsdLI3x+7FeFZnomqgQeKFbii2AgVDgvggBmNRrqrn8wULXH0c7ODXSHLLx1IovJrsos1+hQwRvh32DjyHbgYn15PJlUSPRgfvs71A3ydwD5iJ1UUXLm8AirwRC34V8NRQpX3Hhkg2bB8HZGQjdGNDzNTbehXvdtaYwsJrQ5JyMg3BIfvG3SCSabhwml72YhGOjpJxjvvDwtXO4saHKjen3eDk+a2uU4p4aLy6kd2vkB+fIsQTSqQPZO89aWzrLUSqN+KqH0Ztw6gMynHZRGkL5L9i24DWbAU6fE18h1C2Fk8SWXALCy5B5bbu8PCtrfs17xnYKFgGjhGY9Tcap4TbLJFQodUdUIIG0= ceph-65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:40:44.186 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoL1KgqdJMuGGLulpOEU7LwfXPy53zHBCgCzcvFID5VFnGDqUi8cN3YBuAaqQqd9NeYAttZxP/GK6gXC+qlvQtRe5fAMKIDUjdipZEuLYLD+XZXyCnSyppbl2zbF5xIeRybMDroySIYYF7rOBmz6QXfKpnq3+hsdLI3x+7FeFZnomqgQeKFbii2AgVDgvggBmNRrqrn8wULXH0c7ODXSHLLx1IovJrsos1+hQwRvh32DjyHbgYn15PJlUSPRgfvs71A3ydwD5iJ1UUXLm8AirwRC34V8NRQpX3Hhkg2bB8HZGQjdGNDzNTbehXvdtaYwsJrQ5JyMg3BIfvG3SCSabhwml72YhGOjpJxjvvDwtXO4saHKjen3eDk+a2uU4p4aLy6kd2vkB+fIsQTSqQPZO89aWzrLUSqN+KqH0Ztw6gMynHZRGkL5L9i24DWbAU6fE18h1C2Fk8SWXALCy5B5bbu8PCtrfs17xnYKFgGjhGY9Tcap4TbLJFQodUdUIIG0= ceph-65498394-c7d6-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:40:44.233 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoL1KgqdJMuGGLulpOEU7LwfXPy53zHBCgCzcvFID5VFnGDqUi8cN3YBuAaqQqd9NeYAttZxP/GK6gXC+qlvQtRe5fAMKIDUjdipZEuLYLD+XZXyCnSyppbl2zbF5xIeRybMDroySIYYF7rOBmz6QXfKpnq3+hsdLI3x+7FeFZnomqgQeKFbii2AgVDgvggBmNRrqrn8wULXH0c7ODXSHLLx1IovJrsos1+hQwRvh32DjyHbgYn15PJlUSPRgfvs71A3ydwD5iJ1UUXLm8AirwRC34V8NRQpX3Hhkg2bB8HZGQjdGNDzNTbehXvdtaYwsJrQ5JyMg3BIfvG3SCSabhwml72YhGOjpJxjvvDwtXO4saHKjen3eDk+a2uU4p4aLy6kd2vkB+fIsQTSqQPZO89aWzrLUSqN+KqH0Ztw6gMynHZRGkL5L9i24DWbAU6fE18h1C2Fk8SWXALCy5B5bbu8PCtrfs17xnYKFgGjhGY9Tcap4TbLJFQodUdUIIG0= ceph-65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:40:44.340 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoL1KgqdJMuGGLulpOEU7LwfXPy53zHBCgCzcvFID5VFnGDqUi8cN3YBuAaqQqd9NeYAttZxP/GK6gXC+qlvQtRe5fAMKIDUjdipZEuLYLD+XZXyCnSyppbl2zbF5xIeRybMDroySIYYF7rOBmz6QXfKpnq3+hsdLI3x+7FeFZnomqgQeKFbii2AgVDgvggBmNRrqrn8wULXH0c7ODXSHLLx1IovJrsos1+hQwRvh32DjyHbgYn15PJlUSPRgfvs71A3ydwD5iJ1UUXLm8AirwRC34V8NRQpX3Hhkg2bB8HZGQjdGNDzNTbehXvdtaYwsJrQ5JyMg3BIfvG3SCSabhwml72YhGOjpJxjvvDwtXO4saHKjen3eDk+a2uU4p4aLy6kd2vkB+fIsQTSqQPZO89aWzrLUSqN+KqH0Ztw6gMynHZRGkL5L9i24DWbAU6fE18h1C2Fk8SWXALCy5B5bbu8PCtrfs17xnYKFgGjhGY9Tcap4TbLJFQodUdUIIG0= ceph-65498394-c7d6-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-10T05:40:44.387 INFO:teuthology.orchestra.run.smithi110.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwoL1KgqdJMuGGLulpOEU7LwfXPy53zHBCgCzcvFID5VFnGDqUi8cN3YBuAaqQqd9NeYAttZxP/GK6gXC+qlvQtRe5fAMKIDUjdipZEuLYLD+XZXyCnSyppbl2zbF5xIeRybMDroySIYYF7rOBmz6QXfKpnq3+hsdLI3x+7FeFZnomqgQeKFbii2AgVDgvggBmNRrqrn8wULXH0c7ODXSHLLx1IovJrsos1+hQwRvh32DjyHbgYn15PJlUSPRgfvs71A3ydwD5iJ1UUXLm8AirwRC34V8NRQpX3Hhkg2bB8HZGQjdGNDzNTbehXvdtaYwsJrQ5JyMg3BIfvG3SCSabhwml72YhGOjpJxjvvDwtXO4saHKjen3eDk+a2uU4p4aLy6kd2vkB+fIsQTSqQPZO89aWzrLUSqN+KqH0Ztw6gMynHZRGkL5L9i24DWbAU6fE18h1C2Fk8SWXALCy5B5bbu8PCtrfs17xnYKFgGjhGY9Tcap4TbLJFQodUdUIIG0= ceph-65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:40:44.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-10T05:40:44.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:44 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/1258206815' entity='client.admin' 2024-02-10T05:40:44.638 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:40:47.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:46 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:47.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:46 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:47.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:46 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:40:47.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:46 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:47.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:46 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:47.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:46 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:47.661 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-10T05:40:47.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-10T05:40:47.938 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:40:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:48 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3462672681' entity='client.admin' 2024-02-10T05:40:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:48 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:48.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:48 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:50.597 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi105 2024-02-10T05:40:50.597 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:40:50.597 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.conf 2024-02-10T05:40:50.615 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:40:50.615 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:40:50.672 INFO:tasks.cephadm:Adding host smithi105 to orchestrator... 2024-02-10T05:40:50.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch host add smithi105 2024-02-10T05:40:50.908 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:51.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:51.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:51 smithi029 ceph-mon[25692]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:40:51.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:51 smithi029 ceph-mon[25692]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:40:53.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:53 smithi029 ceph-mon[25692]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:40:54.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:54 smithi029 ceph-mon[25692]: Deploying cephadm binary to smithi105 2024-02-10T05:40:56.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:55 smithi029 ceph-mon[25692]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:40:57.232 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi105' with addr '172.21.15.105' 2024-02-10T05:40:57.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:57 smithi029 ceph-mon[25692]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:40:57.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:57 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:57.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:57 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:40:57.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:57 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:40:57.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T05:40:58.192 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:40:58.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:58 smithi029 ceph-mon[25692]: Added host smithi105 2024-02-10T05:40:58.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:58 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:00.015 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:41:00.015 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}] 2024-02-10T05:41:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:40:59 smithi029 ceph-mon[25692]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:00.615 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi110 2024-02-10T05:41:00.615 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:41:00.616 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.conf 2024-02-10T05:41:00.634 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:41:00.634 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:00.691 INFO:tasks.cephadm:Adding host smithi110 to orchestrator... 2024-02-10T05:41:00.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch host add smithi110 2024-02-10T05:41:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:00 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:00 smithi029 ceph-mon[25692]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:41:00.928 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:41:02.026 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:01 smithi029 ceph-mon[25692]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:04.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:04.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:04.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: Updating smithi105:/etc/ceph/ceph.conf 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:04.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:05.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:05 smithi029 ceph-mon[25692]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:05.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:05 smithi029 ceph-mon[25692]: Deploying cephadm binary to smithi110 2024-02-10T05:41:05.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:05 smithi029 ceph-mon[25692]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:05.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:05 smithi029 ceph-mon[25692]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:41:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:06 smithi029 ceph-mon[25692]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:07.095 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi110' with addr '172.21.15.110' 2024-02-10T05:41:07.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-10T05:41:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:41:08.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:08 smithi029 ceph-mon[25692]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:08.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:08 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:08.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:08 smithi029 ceph-mon[25692]: Added host smithi110 2024-02-10T05:41:08.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:08 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:08.364 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:08 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:09.272 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:09 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:09.707 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:41:09.707 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.110", "hostname": "smithi110", "labels": [], "status": ""}] 2024-02-10T05:41:10.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:10 smithi029 ceph-mon[25692]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:10.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:10 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:10.329 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-10T05:41:10.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-10T05:41:10.568 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:41:11.379 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:11 smithi029 ceph-mon[25692]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:41:12.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:12 smithi029 ceph-mon[25692]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:13.119 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-02-10T05:41:13.341 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3361299865' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:41:13.341 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:13.341 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:13.341 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:13.341 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:13.342 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:41:13.342 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:13.342 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:13.768 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-02-10T05:41:13.768 INFO:tasks.cephadm:Adding mon.b on smithi105 2024-02-10T05:41:13.768 INFO:tasks.cephadm:Adding mon.c on smithi110 2024-02-10T05:41:13.768 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c' 2024-02-10T05:41:13.994 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:14.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:14.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: Updating smithi110:/etc/ceph/ceph.conf 2024-02-10T05:41:14.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:14.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3361299865' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:41:14.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:14.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:14.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:14.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:15.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:15 smithi029 ceph-mon[25692]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:15.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:15 smithi029 ceph-mon[25692]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:41:15.619 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled mon update... 2024-02-10T05:41:16.199 DEBUG:teuthology.orchestra.run.smithi105:mon.b> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.b.service 2024-02-10T05:41:16.202 DEBUG:teuthology.orchestra.run.smithi110:mon.c> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.c.service 2024-02-10T05:41:16.204 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:41:16.204 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:41:16.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:16.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:16.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:16.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:16.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:16.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:16.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:16.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:16.944 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:17.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:17 smithi029 ceph-mon[25692]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:17.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:17 smithi029 ceph-mon[25692]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c;count:3 2024-02-10T05:41:17.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:17 smithi029 ceph-mon[25692]: Deploying daemon mon.c on smithi110 2024-02-10T05:41:18.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:18 smithi029 ceph-mon[25692]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:20.198 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:41:20.198 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":1,"fsid":"65498394-c7d6-11ee-95b8-87774f69a715","modified":"2024-02-10T05:39:34.507354Z","created":"2024-02-10T05:39:34.507354Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-10T05:41:20.203 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 1 2024-02-10T05:41:20.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:20 smithi029 ceph-mon[25692]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:20.488 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:20 smithi110 systemd[1]: Starting Ceph mon.c for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:41:21.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 podman[30831]: 2024-02-10 05:41:20.948446082 +0000 UTC m=+0.018401531 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:41:21.162 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 podman[30831]: 2024-02-10 05:41:21.056050156 +0000 UTC m=+0.126005592 container create 57ca5780fbb9144f1bbe5befd8780069e3412970eedeeb678865c08c70a6ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T05:41:21.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:21 smithi029 ceph-mon[25692]: from='client.? 172.21.15.110:0/1046661574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:41:21.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 podman[30831]: 2024-02-10 05:41:21.250297279 +0000 UTC m=+0.320252725 container init 57ca5780fbb9144f1bbe5befd8780069e3412970eedeeb678865c08c70a6ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T05:41:21.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 podman[30831]: 2024-02-10 05:41:21.255796854 +0000 UTC m=+0.325752291 container start 57ca5780fbb9144f1bbe5befd8780069e3412970eedeeb678865c08c70a6ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-10T05:41:21.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:41:21.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 7 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pidfile_write: ignore empty --pid-file 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: load: jerasure load: lrc 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Git sha 0 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: DB SUMMARY 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: DB Session ID: V9XWIBYABHLO65M520M9 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-10T05:41:21.743 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.env: 0x558ebe1b0ae0 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.info_log: 0x558ebf4d18c0 2024-02-10T05:41:21.744 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.statistics: (nil) 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.use_fsync: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:41:21.745 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.db_log_dir: 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.wal_dir: 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:41:21.746 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.write_buffer_manager: 0x558ebf4a5400 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:41:21.747 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.unordered_write: 0 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.row_cache: None 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.wal_filter: None 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:41:21.748 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.wal_compression: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:41:21.749 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:41:21.750 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_open_files: -1 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Compression algorithms supported: 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kZSTD supported: 0 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:41:21.751 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.merge_operator: 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_filter: None 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:41:21.752 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558ebf4d19c0) 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: index_type: 0 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: data_block_index_type: 0 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: index_shortening: 1 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:41:21.753 INFO:journalctl@ceph.mon.c.smithi110.stdout: checksum: 4 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: no_block_cache: 0 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache: 0x558ebf4d5090 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache_options: 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: capacity : 536870912 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: num_shard_bits : 4 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: strict_capacity_limit : 0 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_cache_compressed: (nil) 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: persistent_cache: (nil) 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_size: 4096 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_size_deviation: 10 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_restart_interval: 16 2024-02-10T05:41:21.754 INFO:journalctl@ceph.mon.c.smithi110.stdout: index_block_restart_interval: 1 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: metadata_block_size: 4096 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: partition_filters: 0 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: use_delta_encoding: 1 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: filter_policy: bloomfilter 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: whole_key_filtering: 1 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: verify_compression: 0 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: format_version: 5 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: enable_index_compression: 1 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: block_align: 0 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: max_auto_readahead_size: 262144 2024-02-10T05:41:21.755 INFO:journalctl@ceph.mon.c.smithi110.stdout: prepopulate_block_cache: 0 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression: NoCompression 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.num_levels: 7 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:41:21.756 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:41:21.757 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:41:21.758 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:41:21.759 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:41:21.760 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:41:21.761 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.ttl: 2592000 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:41:21.762 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:41:21.763 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:41:21.763 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-10T05:41:21.764 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 73746d82-01b4-419c-acd4-71ba9087713a 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543681295949, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543681296430, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707543681, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "73746d82-01b4-419c-acd4-71ba9087713a", "db_session_id": "V9XWIBYABHLO65M520M9", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543681296538, "job": 1, "event": "recovery_finished"} 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558ebf5c6000 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: DB pointer 0x558ebf5b0000 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** DB Stats ** 2024-02-10T05:41:21.765 INFO:journalctl@ceph.mon.c.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** Compaction Stats [default] ** 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:21.766 INFO:journalctl@ceph.mon.c.smithi110.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** Compaction Stats [default] ** 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2024-02-10T05:41:21.767 INFO:journalctl@ceph.mon.c.smithi110.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: Block cache BinnedLRUCache@0x558ebf4d5090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-02-10T05:41:21.768 INFO:journalctl@ceph.mon.c.smithi110.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: using public_addr v2:172.21.15.110:0/0 -> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: starting mon.c rank -1 at public addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] at bind addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(???) e0 preinit fsid 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).mds e1 new map 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).mds e1 print_map 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout: e1 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout: legacy client fscid: -1 2024-02-10T05:41:21.769 INFO:journalctl@ceph.mon.c.smithi110.stdout: 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout: No filesystems configured 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:41:21.770 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Added host smithi105 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:21.771 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi105:/etc/ceph/ceph.conf 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Deploying cephadm binary to smithi110 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.772 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Added host smithi110 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3361299865' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.773 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi110:/etc/ceph/ceph.conf 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3361299865' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.774 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 bash[30831]: 57ca5780fbb9144f1bbe5befd8780069e3412970eedeeb678865c08c70a6ee99 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:21.775 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c;count:3 2024-02-10T05:41:21.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: Deploying daemon mon.c on smithi110 2024-02-10T05:41:21.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:21.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: from='client.? 172.21.15.110:0/1046661574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:41:21.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 ceph-mon[30869]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T05:41:21.776 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:21 smithi110 systemd[1]: Started Ceph mon.c for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:41:21.849 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:41:21.849 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:41:22.082 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:41:26.707 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:41:26.707 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":2,"fsid":"65498394-c7d6-11ee-95b8-87774f69a715","modified":"2024-02-10T05:41:21.320522Z","created":"2024-02-10T05:39:34.507354Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:41:26.707 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: mon.a calling monitor election 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: mon.c calling monitor election 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.708 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: fsmap 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: mgrmap e13: y(active, since 52s) 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: overall HEALTH_OK 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:26.709 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:26 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:26.709 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 2 2024-02-10T05:41:26.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:41:26.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: mon.a calling monitor election 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: mon.c calling monitor election 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:26.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: fsmap 2024-02-10T05:41:26.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:26.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: mgrmap e13: y(active, since 52s) 2024-02-10T05:41:26.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: overall HEALTH_OK 2024-02-10T05:41:26.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:26.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:26 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:27.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:27 smithi110 ceph-mon[30869]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:27.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:27 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:27.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:27 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:27.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:27 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:27.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:27 smithi110 ceph-mon[30869]: Deploying daemon mon.b on smithi105 2024-02-10T05:41:27.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:27 smithi110 ceph-mon[30869]: from='client.? 172.21.15.110:0/973035551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:41:27.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:27 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:27.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:27 smithi029 ceph-mon[25692]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:27.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:27 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:27.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:27 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:27.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:27 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:27.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:27 smithi029 ceph-mon[25692]: Deploying daemon mon.b on smithi105 2024-02-10T05:41:27.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:27 smithi029 ceph-mon[25692]: from='client.? 172.21.15.110:0/973035551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:41:27.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:27 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:28.264 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:41:28.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:41:28.511 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:41:29.700 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:29 smithi110 ceph-mon[30869]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:29.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:29 smithi029 ceph-mon[25692]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.285 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 podman[29817]: 2024-02-10 05:41:30.091252712 +0000 UTC m=+0.214634253 container create 01e2d35d4f7afa83895302f4e47ff3ba2166453cb32f98e6e1b62b6c02ffcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T05:41:30.286 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 podman[29817]: 2024-02-10 05:41:30.285318705 +0000 UTC m=+0.408700250 container init 01e2d35d4f7afa83895302f4e47ff3ba2166453cb32f98e6e1b62b6c02ffcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-10T05:41:30.319 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:41:30.319 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":2,"fsid":"65498394-c7d6-11ee-95b8-87774f69a715","modified":"2024-02-10T05:41:21.320522Z","created":"2024-02-10T05:39:34.507354Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:41:30.321 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 2 2024-02-10T05:41:30.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 podman[29817]: 2024-02-10 05:41:30.290684503 +0000 UTC m=+0.414066044 container start 01e2d35d4f7afa83895302f4e47ff3ba2166453cb32f98e6e1b62b6c02ffcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-10T05:41:30.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: set uid:gid to 167:167 (ceph:ceph) 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 7 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pidfile_write: ignore empty --pid-file 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: load: jerasure load: lrc 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: RocksDB version: 7.9.2 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Git sha 0 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: DB SUMMARY 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: DB Session ID: NPL1QNRS9EUC8E1N6DWV 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: CURRENT file: CURRENT 2024-02-10T05:41:30.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: IDENTITY file: IDENTITY 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.error_if_exists: 0 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.create_if_missing: 0 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.paranoid_checks: 1 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-10T05:41:30.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.env: 0x559535854ae0 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.fs: PosixFileSystem 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.info_log: 0x5595362898c0 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_file_opening_threads: 16 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.statistics: (nil) 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.use_fsync: 0 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_log_file_size: 0 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.keep_log_file_num: 1000 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.recycle_log_file_num: 0 2024-02-10T05:41:30.663 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.allow_fallocate: 1 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.allow_mmap_reads: 0 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.allow_mmap_writes: 0 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.use_direct_reads: 0 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.create_missing_column_families: 0 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.db_log_dir: 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.wal_dir: 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-10T05:41:30.664 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.advise_random_on_open: 1 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.db_write_buffer_size: 0 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.write_buffer_manager: 0x55953625d400 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.rate_limiter: (nil) 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.wal_recovery_mode: 2 2024-02-10T05:41:30.665 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.enable_thread_tracking: 0 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.enable_pipelined_write: 0 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.unordered_write: 0 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.row_cache: None 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.wal_filter: None 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.allow_ingest_behind: 0 2024-02-10T05:41:30.666 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.two_write_queues: 0 2024-02-10T05:41:30.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.manual_wal_flush: 0 2024-02-10T05:41:30.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.wal_compression: 0 2024-02-10T05:41:30.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.atomic_flush: 0 2024-02-10T05:41:30.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-10T05:41:30.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-10T05:41:30.667 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.log_readahead_size: 0 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.best_efforts_recovery: 0 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.allow_data_in_errors: 0 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.db_host_id: __hostname__ 2024-02-10T05:41:30.669 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_background_jobs: 2 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_background_compactions: -1 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_subcompactions: 1 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_total_wal_size: 0 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_open_files: -1 2024-02-10T05:41:30.670 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bytes_per_sync: 0 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_readahead_size: 0 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_background_flushes: -1 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Compression algorithms supported: 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kZSTD supported: 0 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kXpressCompression supported: 0 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kLZ4HCCompression supported: 1 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kZlibCompression supported: 1 2024-02-10T05:41:30.671 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kSnappyCompression supported: 1 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kLZ4Compression supported: 1 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: kBZip2Compression supported: 0 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.merge_operator: 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_filter: None 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_filter_factory: None 2024-02-10T05:41:30.672 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.sst_partitioner_factory: None 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5595362899c0) 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: cache_index_and_filter_blocks: 1 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: pin_top_level_index_and_filter: 1 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_type: 0 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: data_block_index_type: 0 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_shortening: 1 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-10T05:41:30.673 INFO:journalctl@ceph.mon.b.smithi105.stdout: checksum: 4 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: no_block_cache: 0 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache: 0x55953628d090 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_name: BinnedLRUCache 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_options: 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: capacity : 536870912 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: num_shard_bits : 4 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: strict_capacity_limit : 0 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: high_pri_pool_ratio: 0.000 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_cache_compressed: (nil) 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: persistent_cache: (nil) 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_size: 4096 2024-02-10T05:41:30.674 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_size_deviation: 10 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_restart_interval: 16 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: index_block_restart_interval: 1 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: metadata_block_size: 4096 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: partition_filters: 0 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: use_delta_encoding: 1 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: filter_policy: bloomfilter 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: whole_key_filtering: 1 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: verify_compression: 0 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: read_amp_bytes_per_bit: 0 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: format_version: 5 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: enable_index_compression: 1 2024-02-10T05:41:30.675 INFO:journalctl@ceph.mon.b.smithi105.stdout: block_align: 0 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout: max_auto_readahead_size: 262144 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout: prepopulate_block_cache: 0 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout: initial_auto_readahead_size: 8192 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout: num_file_reads_for_auto_readahead: 2 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.write_buffer_size: 33554432 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_write_buffer_number: 2 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression: NoCompression 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression: Disabled 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.prefix_extractor: nullptr 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-10T05:41:30.676 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.num_levels: 7 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-10T05:41:30.677 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.level: 32767 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.strategy: 0 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.enabled: false 2024-02-10T05:41:30.678 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.target_file_size_base: 67108864 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-10T05:41:30.679 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.arena_block_size: 1048576 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-10T05:41:30.680 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.disable_auto_compactions: 0 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-10T05:41:30.681 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.table_properties_collectors: 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.inplace_update_support: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.bloom_locality: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.max_successive_merges: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.paranoid_file_checks: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.force_consistency_checks: 1 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.report_bg_io_stats: 0 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.ttl: 2592000 2024-02-10T05:41:30.682 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.enable_blob_files: false 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.min_blob_size: 0 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.blob_file_size: 268435456 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-10T05:41:30.683 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.blob_file_starting_level: 0 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e4b668da-108f-41ec-aa49-359dadda2ec0 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543690331963, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543690332374, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707543690, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e4b668da-108f-41ec-aa49-359dadda2ec0", "db_session_id": "NPL1QNRS9EUC8E1N6DWV", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707543690332468, "job": 1, "event": "recovery_finished"} 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55953637e000 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: DB pointer 0x559536368000 2024-02-10T05:41:30.684 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** DB Stats ** 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-10T05:41:30.685 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** Compaction Stats [default] ** 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** Compaction Stats [default] ** 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-10T05:41:30.686 INFO:journalctl@ceph.mon.b.smithi105.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: Cumulative compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:41:30.687 INFO:journalctl@ceph.mon.b.smithi105.stdout: Interval compaction: 0.00 GB write, 0.12 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout: Block cache BinnedLRUCache@0x55953628d090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: using public_addr v2:172.21.15.105:0/0 -> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: starting mon.b rank -1 at public addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] at bind addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(???) e0 preinit fsid 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).mds e1 new map 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-10T05:41:30.688 INFO:journalctl@ceph.mon.b.smithi105.stdout: e1 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout: legacy client fscid: -1 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout: 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout: No filesystems configured 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-10T05:41:30.689 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Added host smithi105 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.690 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi110", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi105:/etc/ceph/ceph.conf 2024-02-10T05:41:30.691 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Deploying cephadm binary to smithi110 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Added host smithi110 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:30.692 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3361299865' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi110", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:41:30.693 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi110:/etc/ceph/ceph.conf 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3361299865' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:30.694 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c;count:3 2024-02-10T05:41:30.695 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Deploying daemon mon.c on smithi110 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.? 172.21.15.110:0/1046661574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.a calling monitor election 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.c calling monitor election 2024-02-10T05:41:30.696 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: fsmap 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mgrmap e13: y(active, since 52s) 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: overall HEALTH_OK 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.697 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: Deploying daemon mon.b on smithi105 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='client.? 172.21.15.110:0/973035551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 ceph-mon[29854]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 bash[29817]: 01e2d35d4f7afa83895302f4e47ff3ba2166453cb32f98e6e1b62b6c02ffcdba 2024-02-10T05:41:30.698 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:30 smithi105 systemd[1]: Started Ceph mon.b for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:41:31.923 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-10T05:41:31.923 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-10T05:41:32.164 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: mon.c calling monitor election 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: mon.a calling monitor election 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: fsmap 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: mgrmap e13: y(active, since 61s) 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-10T05:41:35.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:41:35.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: mon.b (rank 2) addr [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] is down (out of quorum) 2024-02-10T05:41:35.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:35.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:35.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:35.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:35.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: mon.c calling monitor election 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: mon.a calling monitor election 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:35.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: fsmap 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: mgrmap e13: y(active, since 61s) 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: mon.b (rank 2) addr [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] is down (out of quorum) 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:35.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:36.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:36.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:36.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:36.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.041 INFO:teuthology.orchestra.run.smithi110.stdout: 2024-02-10T05:41:37.041 INFO:teuthology.orchestra.run.smithi110.stdout:{"epoch":3,"fsid":"65498394-c7d6-11ee-95b8-87774f69a715","modified":"2024-02-10T05:41:30.354703Z","created":"2024-02-10T05:39:34.507354Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:3300","nonce":0},{"type":"v1","addr":"172.21.15.110:6789","nonce":0}]},"addr":"172.21.15.110:6789/0","public_addr":"172.21.15.110:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-10T05:41:37.044 INFO:teuthology.orchestra.run.smithi110.stderr:dumped monmap epoch 3 2024-02-10T05:41:37.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: mon.c calling monitor election 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: mon.a calling monitor election 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: fsmap 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: mgrmap e13: y(active, since 61s) 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: mon.b (rank 2) addr [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] is down (out of quorum) 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:37.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:37.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:37.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:37.760 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-10T05:41:37.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-10T05:41:38.014 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:41:38.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: mon.b calling monitor election 2024-02-10T05:41:38.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: mon.b calling monitor election 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: mon.c calling monitor election 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: mon.a calling monitor election 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: fsmap 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: mgrmap e13: y(active, since 63s) 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: Cluster is now healthy 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: overall HEALTH_OK 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:38.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:38 smithi029 ceph-mon[25692]: Reconfiguring daemon mon.a on smithi029 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: mon.b calling monitor election 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: mon.b calling monitor election 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: mon.c calling monitor election 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: mon.a calling monitor election 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: fsmap 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: mgrmap e13: y(active, since 63s) 2024-02-10T05:41:38.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: Cluster is now healthy 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: overall HEALTH_OK 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:41:38.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:38.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:38.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:38.662 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:38 smithi105 ceph-mon[29854]: Reconfiguring daemon mon.a on smithi029 2024-02-10T05:41:38.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: mon.b calling monitor election 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: mon.b calling monitor election 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: mon.c calling monitor election 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: mon.a calling monitor election 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],c=[v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: fsmap 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: osdmap e4: 0 total, 0 up, 0 in 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: mgrmap e13: y(active, since 63s) 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-02-10T05:41:38.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: Cluster is now healthy 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: overall HEALTH_OK 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: Reconfiguring mon.a (unknown last config time)... 2024-02-10T05:41:38.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:38.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:38.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:38.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:38 smithi110 ceph-mon[30869]: Reconfiguring daemon mon.a on smithi029 2024-02-10T05:41:39.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:39 smithi029 ceph-mon[25692]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:39.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:39 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:39.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:39 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:39.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:39 smithi105 ceph-mon[29854]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:39.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:39 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:39.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:39 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:39.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:39 smithi110 ceph-mon[30869]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:39.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:39 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:41:39.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:39 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:40.844 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:41:40.845 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-02-10T05:41:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:41:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2024-02-10T05:41:40.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:40.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:40.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:40 smithi105 ceph-mon[29854]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:41:40.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:40.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:40.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:40.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:40 smithi105 ceph-mon[29854]: Reconfiguring daemon mon.b on smithi105 2024-02-10T05:41:40.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:40.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:40.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:40 smithi110 ceph-mon[30869]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:41:40.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:40.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:40.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:40.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:40 smithi110 ceph-mon[30869]: Reconfiguring daemon mon.b on smithi105 2024-02-10T05:41:41.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:41.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:41.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:40 smithi029 ceph-mon[25692]: Reconfiguring mon.b (monmap changed)... 2024-02-10T05:41:41.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:41.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:41.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:41.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:40 smithi029 ceph-mon[25692]: Reconfiguring daemon mon.b on smithi105 2024-02-10T05:41:41.478 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-10T05:41:41.478 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:41:41.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:41:41.509 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:41:41.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:41.577 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:41:41.577 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:41:41.609 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:41:41.609 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:41.677 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:41:41.677 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.conf 2024-02-10T05:41:41.705 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:41:41.706 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:41:41.772 INFO:tasks.cephadm:Adding mgr.y on smithi029 2024-02-10T05:41:41.773 INFO:tasks.cephadm:Adding mgr.x on smithi105 2024-02-10T05:41:41.773 INFO:tasks.cephadm:Adding mgr.z on smithi110 2024-02-10T05:41:41.773 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch apply mgr '3;smithi029=y;smithi105=x;smithi110=z' 2024-02-10T05:41:41.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:41 smithi105 ceph-mon[29854]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:41.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:41 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2472485163' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:41.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:41 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:41.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:41 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:41.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:41 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:41.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:41 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:41.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:41 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:41.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:41 smithi110 ceph-mon[30869]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:41.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:41 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2472485163' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:41.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:41 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:41.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:41 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:41.971 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:41 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:41.972 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:41 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:41.972 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:41 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:42.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:41 smithi029 ceph-mon[25692]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:42.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:41 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2472485163' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:42.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:41 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:42.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:41 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:42.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:41 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-10T05:41:42.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:41 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-10T05:41:42.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:41 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:42.738 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:41:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:42 smithi029 ceph-mon[25692]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:41:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:42 smithi029 ceph-mon[25692]: Reconfiguring daemon mon.c on smithi110 2024-02-10T05:41:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:42 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:42 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:42.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:42 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:43.101 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:42 smithi110 ceph-mon[30869]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:41:43.101 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:42 smithi110 ceph-mon[30869]: Reconfiguring daemon mon.c on smithi110 2024-02-10T05:41:43.101 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:42 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:43.101 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:42 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:43.101 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:42 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:43.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:42 smithi105 ceph-mon[29854]: Reconfiguring mon.c (monmap changed)... 2024-02-10T05:41:43.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:42 smithi105 ceph-mon[29854]: Reconfiguring daemon mon.c on smithi110 2024-02-10T05:41:43.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:42 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:43.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:42 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:43.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:42 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:43.862 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:43 smithi110 ceph-mon[30869]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:43.863 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:43 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:43.863 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:43 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:43.863 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:43 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:44.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:43 smithi029 ceph-mon[25692]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:44.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:43 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:44.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:43 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:44.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:43 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:44.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:43 smithi105 ceph-mon[29854]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:44.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:43 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:44.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:43 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:44.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:43 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:44.542 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled mgr update... 2024-02-10T05:41:45.179 DEBUG:teuthology.orchestra.run.smithi105:mgr.x> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.x.service 2024-02-10T05:41:45.182 DEBUG:teuthology.orchestra.run.smithi110:mgr.z> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.z.service 2024-02-10T05:41:45.184 INFO:tasks.cephadm:Deploying OSDs... 2024-02-10T05:41:45.184 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:41:45.184 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:41:45.203 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:41:45.203 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/nvme1n1 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/nvme1n1 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:40:44.836412453 +0000 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:49.092278327 +0000 2024-02-10T05:41:45.260 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:49.092278327 +0000 2024-02-10T05:41:45.261 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:49.082278489 +0000 2024-02-10T05:41:45.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-10T05:41:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:41:45.330 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:41:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000151009 s, 3.4 MB/s 2024-02-10T05:41:45.332 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-10T05:41:45.389 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/nvme2n1 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/nvme2n1 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:40:44.845412306 +0000 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:49.233276030 +0000 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:49.233276030 +0000 2024-02-10T05:41:45.446 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:49.225276160 +0000 2024-02-10T05:41:45.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-10T05:41:45.514 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:41:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:41:45.515 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000141938 s, 3.6 MB/s 2024-02-10T05:41:45.516 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-10T05:41:45.573 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/nvme3n1 2024-02-10T05:41:45.629 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/nvme3n1 2024-02-10T05:41:45.629 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:45.629 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 788 Links: 1 Device type: 103,6 2024-02-10T05:41:45.630 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:45.630 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:45.630 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:40:44.853412176 +0000 2024-02-10T05:41:45.630 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:49.381273620 +0000 2024-02-10T05:41:45.630 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:49.381273620 +0000 2024-02-10T05:41:45.630 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:49.374273734 +0000 2024-02-10T05:41:45.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-10T05:41:45.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:45.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=y;smithi105=x;smithi110=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:45.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: Saving service mgr spec with placement smithi029=y;smithi105=x;smithi110=z;count:3 2024-02-10T05:41:45.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:45.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:45.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:45.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:45.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:45.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:45.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:41:45.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:45.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:45.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:45 smithi029 ceph-mon[25692]: Deploying daemon mgr.z on smithi110 2024-02-10T05:41:45.661 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:41:45.661 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:41:45.662 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000132012 s, 3.9 MB/s 2024-02-10T05:41:45.663 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-10T05:41:45.719 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/nvme4n1 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=y;smithi105=x;smithi110=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: Saving service mgr spec with placement smithi029=y;smithi105=x;smithi110=z;count:3 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:41:45.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:45.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:45.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:45 smithi110 ceph-mon[30869]: Deploying daemon mgr.z on smithi110 2024-02-10T05:41:45.774 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/nvme4n1 2024-02-10T05:41:45.774 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:45.774 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-02-10T05:41:45.774 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:45.774 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:45.774 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-02-10 05:40:44.862412029 +0000 2024-02-10T05:41:45.774 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-02-10 05:36:49.527271242 +0000 2024-02-10T05:41:45.775 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-02-10 05:36:49.527271242 +0000 2024-02-10T05:41:45.775 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-02-10 05:36:49.517271405 +0000 2024-02-10T05:41:45.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-10T05:41:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-02-10T05:41:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-02-10T05:41:45.840 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000112302 s, 4.6 MB/s 2024-02-10T05:41:45.841 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-10T05:41:45.897 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:41:45.897 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi029=y;smithi105=x;smithi110=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: Saving service mgr spec with placement smithi029=y;smithi105=x;smithi110=z;count:3 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:45.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:45.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:45.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:41:45.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:45.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:45.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:45 smithi105 ceph-mon[29854]: Deploying daemon mgr.z on smithi110 2024-02-10T05:41:45.915 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:41:45.915 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/nvme1n1 2024-02-10T05:41:45.972 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/nvme1n1 2024-02-10T05:41:45.972 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:45.973 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-10T05:41:45.973 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:45.973 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:45.973 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:41:02.074496998 +0000 2024-02-10T05:41:45.973 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:50.740319715 +0000 2024-02-10T05:41:45.973 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:50.740319715 +0000 2024-02-10T05:41:45.973 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:50.729318325 +0000 2024-02-10T05:41:45.973 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-10T05:41:46.041 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:41:46.041 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:41:46.041 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000260462 s, 2.0 MB/s 2024-02-10T05:41:46.043 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-10T05:41:46.101 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/nvme2n1 2024-02-10T05:41:46.157 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/nvme2n1 2024-02-10T05:41:46.157 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:46.157 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-10T05:41:46.157 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:46.157 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:46.158 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:41:02.085496785 +0000 2024-02-10T05:41:46.158 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:50.895339303 +0000 2024-02-10T05:41:46.158 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:50.895339303 +0000 2024-02-10T05:41:46.158 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:50.883337787 +0000 2024-02-10T05:41:46.158 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-10T05:41:46.224 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:41:46.225 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:41:46.225 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000230425 s, 2.2 MB/s 2024-02-10T05:41:46.226 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-10T05:41:46.282 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/nvme3n1 2024-02-10T05:41:46.337 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/nvme3n1 2024-02-10T05:41:46.337 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:46.337 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 786 Links: 1 Device type: 103,6 2024-02-10T05:41:46.337 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:46.337 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:46.338 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:41:02.096496571 +0000 2024-02-10T05:41:46.338 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:51.043358007 +0000 2024-02-10T05:41:46.338 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:51.043358007 +0000 2024-02-10T05:41:46.338 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:51.031356490 +0000 2024-02-10T05:41:46.338 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-10T05:41:46.403 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:41:46.403 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:41:46.403 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000191506 s, 2.7 MB/s 2024-02-10T05:41:46.405 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-10T05:41:46.461 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/nvme4n1 2024-02-10T05:41:46.516 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/nvme4n1 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 800 Links: 1 Device type: 103,8 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-02-10 05:41:02.106496377 +0000 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-02-10 05:36:51.195366223 +0000 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-02-10 05:36:51.195366223 +0000 2024-02-10T05:41:46.517 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-02-10 05:36:51.185366418 +0000 2024-02-10T05:41:46.517 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-10T05:41:46.582 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-02-10T05:41:46.582 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-02-10T05:41:46.582 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000193651 s, 2.6 MB/s 2024-02-10T05:41:46.583 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-10T05:41:46.639 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:41:46.639 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2024-02-10T05:41:46.655 DEBUG:teuthology.misc:devs=['/dev/nvme1n1', '/dev/nvme2n1', '/dev/nvme3n1', '/dev/nvme4n1'] 2024-02-10T05:41:46.655 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme1n1 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme1n1 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 762 Links: 1 Device type: 103,2 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:41:11.798394709 +0000 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:52.384492600 +0000 2024-02-10T05:41:46.710 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:52.384492600 +0000 2024-02-10T05:41:46.711 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:52.374492758 +0000 2024-02-10T05:41:46.711 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme1n1 of=/dev/null count=1 2024-02-10T05:41:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:41:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:41:46.783 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000109384 s, 4.7 MB/s 2024-02-10T05:41:46.784 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme1n1 2024-02-10T05:41:46.842 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme2n1 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme2n1 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 775 Links: 1 Device type: 103,4 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:41:11.807394566 +0000 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:52.535490214 +0000 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:52.535490214 +0000 2024-02-10T05:41:46.899 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:52.528490325 +0000 2024-02-10T05:41:46.900 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme2n1 of=/dev/null count=1 2024-02-10T05:41:46.965 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:41:46.966 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:41:46.966 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000122939 s, 4.2 MB/s 2024-02-10T05:41:46.967 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme2n1 2024-02-10T05:41:47.022 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme3n1 2024-02-10T05:41:47.035 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 podman[33049]: 2024-02-10 05:41:46.918670249 +0000 UTC m=+0.017650900 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:41:47.035 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 podman[33049]: 2024-02-10 05:41:47.035859711 +0000 UTC m=+0.134840347 container create ff4c8b64a0b2330b6faf23d3f8797cd83e53fa0552d13ecc7a8636c1cd66e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-10T05:41:47.049 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme3n1 2024-02-10T05:41:47.049 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:47.049 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 787 Links: 1 Device type: 103,6 2024-02-10T05:41:47.049 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:47.049 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:47.050 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:41:11.815394440 +0000 2024-02-10T05:41:47.050 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:52.690487765 +0000 2024-02-10T05:41:47.050 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:52.690487765 +0000 2024-02-10T05:41:47.050 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:52.680487923 +0000 2024-02-10T05:41:47.050 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme3n1 of=/dev/null count=1 2024-02-10T05:41:47.115 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:41:47.115 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:41:47.115 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000124488 s, 4.1 MB/s 2024-02-10T05:41:47.116 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme3n1 2024-02-10T05:41:47.172 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/nvme4n1 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/nvme4n1 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 0 Blocks: 0 IO Block: 4096 block special file 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 5h/5d Inode: 799 Links: 1 Device type: 103,8 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0660/brw-rw----) Uid: ( 0/ root) Gid: ( 6/ disk) 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:fixed_disk_device_t:s0 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2024-02-10 05:41:11.824394297 +0000 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2024-02-10 05:36:52.848485269 +0000 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2024-02-10 05:36:52.848485269 +0000 2024-02-10T05:41:47.227 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: 2024-02-10 05:36:52.838485427 +0000 2024-02-10T05:41:47.227 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/nvme4n1 of=/dev/null count=1 2024-02-10T05:41:47.292 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2024-02-10T05:41:47.292 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2024-02-10T05:41:47.292 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000121636 s, 4.2 MB/s 2024-02-10T05:41:47.293 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/nvme4n1 2024-02-10T05:41:47.350 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/nvme4n1... 2024-02-10T05:41:47.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-10T05:41:47.547 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 podman[33049]: 2024-02-10 05:41:47.314745992 +0000 UTC m=+0.413726641 container init ff4c8b64a0b2330b6faf23d3f8797cd83e53fa0552d13ecc7a8636c1cd66e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-10T05:41:47.547 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 podman[33049]: 2024-02-10 05:41:47.319854541 +0000 UTC m=+0.418835179 container start ff4c8b64a0b2330b6faf23d3f8797cd83e53fa0552d13ecc7a8636c1cd66e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:41:47.547 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 ceph-mgr[33213]: -- 172.21.15.110:0/586657759 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5642092681a0 con 0x56420923ec00 2024-02-10T05:41:47.547 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 bash[33049]: ff4c8b64a0b2330b6faf23d3f8797cd83e53fa0552d13ecc7a8636c1cd66e91e 2024-02-10T05:41:47.547 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 systemd[1]: Started Ceph mgr.z for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:41:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:41:47.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:47 smithi029 ceph-mon[25692]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:47.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:47 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:47.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:47 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:47.877 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:47 smithi110 ceph-mon[30869]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:47.877 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:47 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:47.877 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:47 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:47.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:47 smithi105 ceph-mon[29854]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:47.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:47 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:47.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:47 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:48.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:47.876+0000 7fe7172cd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:41:48.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:47 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:47.968+0000 7fe7172cd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:41:48.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:48 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:48.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:48 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:48.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:48 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:41:48.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:48 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:48.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:48 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:48.715 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:48 smithi029 ceph-mon[25692]: Deploying daemon mgr.x on smithi105 2024-02-10T05:41:48.829 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:48.829 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:48.830 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:41:48.830 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:48.830 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:48.830 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-mon[30869]: Deploying daemon mgr.x on smithi105 2024-02-10T05:41:48.830 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:48.599+0000 7fe7172cd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:41:48.830 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:48.691+0000 7fe7172cd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:41:48.830 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:48.829+0000 7fe7172cd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:41:48.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:48 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:48.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:48 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:48.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:48 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-10T05:41:48.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:48 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:48.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:48 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:48.842 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:48 smithi105 ceph-mon[29854]: Deploying daemon mgr.x on smithi105 2024-02-10T05:41:49.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:48 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:48.923+0000 7fe7172cd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:41:49.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:49 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:49.004+0000 7fe7172cd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:41:49.548 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:49 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:49.262+0000 7fe7172cd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:41:49.548 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:49 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:49.369+0000 7fe7172cd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:41:49.549 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:49 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:49.454+0000 7fe7172cd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:41:49.549 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:49 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:49.547+0000 7fe7172cd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:41:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/nvme4n1 2024-02-10T05:41:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:41:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-10T05:41:49.565 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-02-10T05:41:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-02-10T05:41:49.565 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00974562 s, 1.1 GB/s 2024-02-10T05:41:49.565 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-02-10T05:41:49.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:49 smithi105 ceph-mon[29854]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:49.659 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:49 smithi105 systemd[1]: Starting Ceph mgr.x for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:41:49.876 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:49 smithi029 ceph-mon[25692]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:49.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:49 smithi110 ceph-mon[30869]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:49.989 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:49 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:49.720+0000 7fe7172cd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:41:50.051 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:49 smithi105 podman[31339]: 2024-02-10 05:41:49.777171485 +0000 UTC m=+0.108099368 container create b53f4d6dc2ba87b50f31df51a01d65364fb406911da5758f4e7b027397b259ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T05:41:50.051 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:49 smithi105 podman[31339]: 2024-02-10 05:41:49.686969901 +0000 UTC m=+0.017897791 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:41:50.051 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:50 smithi105 podman[31339]: 2024-02-10 05:41:50.0213366 +0000 UTC m=+0.352264481 container init b53f4d6dc2ba87b50f31df51a01d65364fb406911da5758f4e7b027397b259ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:41:50.052 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:50 smithi105 podman[31339]: 2024-02-10 05:41:50.026573352 +0000 UTC m=+0.357501238 container start b53f4d6dc2ba87b50f31df51a01d65364fb406911da5758f4e7b027397b259ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-10T05:41:50.052 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:50 smithi105 ceph-mgr[31377]: -- 172.21.15.105:0/4122036502 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a3cac561a0 con 0x55a3cac2cc00 2024-02-10T05:41:50.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi029:/dev/nvme4n1 2024-02-10T05:41:50.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:50 smithi105 bash[31339]: b53f4d6dc2ba87b50f31df51a01d65364fb406911da5758f4e7b027397b259ce 2024-02-10T05:41:50.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:50 smithi105 systemd[1]: Started Ceph mgr.x for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:41:50.414 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:41:50.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:50 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:50.578+0000 7f687d4d2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:41:50.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:50 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:50.673+0000 7f687d4d2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:41:50.989 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:50 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:50.697+0000 7fe7172cd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:41:50.990 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:50 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:50.842+0000 7fe7172cd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:41:51.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:51.353+0000 7f687d4d2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:41:51.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-mon[29854]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:51.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:51.480 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-mon[30869]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:51.480 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.481 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.481 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.481 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.481 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:51.482 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:51.191+0000 7fe7172cd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:41:51.482 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:51.286+0000 7fe7172cd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:41:51.482 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:51.479+0000 7fe7172cd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:41:51.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:51 smithi029 ceph-mon[25692]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:51.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:51 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:51 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:51 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:51 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:51.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:51 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:51.739 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:51.670+0000 7fe7172cd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:41:51.774 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:51.447+0000 7f687d4d2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:41:51.775 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:51.593+0000 7f687d4d2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:41:51.775 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:51.690+0000 7f687d4d2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:41:51.775 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:51 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:51.774+0000 7f687d4d2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:41:52.206 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:51 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:51.951+0000 7fe7172cd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:41:52.206 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:52.205+0000 7fe7172cd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:41:52.318 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:52.032+0000 7f687d4d2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:41:52.318 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:52.139+0000 7f687d4d2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:41:52.318 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:52.224+0000 7f687d4d2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:41:52.318 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:52.317+0000 7f687d4d2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:41:52.489 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:52.437+0000 7fe7172cd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:41:52.600 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:52.491+0000 7f687d4d2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:41:52.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:52.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:52.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: Reconfiguring daemon mgr.y on smithi029 2024-02-10T05:41:52.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:41:52.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:41:52.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:52 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: Reconfiguring daemon mgr.y on smithi029 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:41:52.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:52 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: Reconfiguring mgr.y (unknown last config time)... 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: Reconfiguring daemon mgr.y on smithi029 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:41:52.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:52 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:53.489 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:53.221+0000 7fe7172cd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:41:53.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:53.452+0000 7f687d4d2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:41:53.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:53.583+0000 7f687d4d2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:41:53.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:53.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:53.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:53.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:53.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:53.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:53.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:53.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:53.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:53.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:53.928 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:53 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.989 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:53.784+0000 7fe7172cd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:53.879+0000 7fe7172cd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:41:53.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:53 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:54.196 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:53 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:53.914+0000 7f687d4d2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:41:54.196 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:54.003+0000 7f687d4d2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:41:54.196 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:54.194+0000 7f687d4d2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:41:54.489 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:41:54 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:41:54.091+0000 7fe7172cd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:41:54.657 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:54.377+0000 7f687d4d2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:41:54.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:54 smithi029 ceph-mon[25692]: Standby manager daemon z started 2024-02-10T05:41:54.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:54 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-10T05:41:54.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:54 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:41:54.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:54 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-10T05:41:54.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:54 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:41:54.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:54 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:54.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:54.657+0000 7f687d4d2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:41:54.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-mon[29854]: Standby manager daemon z started 2024-02-10T05:41:54.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-10T05:41:54.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:41:54.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-10T05:41:54.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:41:54.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:54.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:54 smithi110 ceph-mon[30869]: Standby manager daemon z started 2024-02-10T05:41:54.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:54 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-10T05:41:54.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:54 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:41:54.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:54 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-10T05:41:54.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:54 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/3068618081' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:41:54.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:54 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:41:55.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:54 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:54.914+0000 7f687d4d2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:41:55.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:55 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:55.146+0000 7f687d4d2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:41:55.937 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:55 smithi105 ceph-mon[29854]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:55.937 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:55 smithi105 ceph-mon[29854]: mgrmap e14: y(active, since 80s), standbys: z 2024-02-10T05:41:55.937 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:55 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-10T05:41:55.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:55 smithi110 ceph-mon[30869]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:55.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:55 smithi110 ceph-mon[30869]: mgrmap e14: y(active, since 80s), standbys: z 2024-02-10T05:41:55.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:55 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-10T05:41:56.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:55 smithi029 ceph-mon[25692]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:56.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:55 smithi029 ceph-mon[25692]: mgrmap e14: y(active, since 80s), standbys: z 2024-02-10T05:41:56.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:55 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-10T05:41:56.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:55 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:55.936+0000 7f687d4d2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:41:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:56 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3238371099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abd76ef6-9f8b-4265-8fb7-b357fd8d1e28"}]: dispatch 2024-02-10T05:41:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:56 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3238371099' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abd76ef6-9f8b-4265-8fb7-b357fd8d1e28"}]': finished 2024-02-10T05:41:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:56 smithi105 ceph-mon[29854]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:41:56.807 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:56 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:41:56.808 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:56 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:56.498+0000 7f687d4d2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:41:56.808 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:56 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:56.595+0000 7f687d4d2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:41:56.808 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:41:56 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:41:56.806+0000 7f687d4d2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:41:56.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:56 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3238371099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abd76ef6-9f8b-4265-8fb7-b357fd8d1e28"}]: dispatch 2024-02-10T05:41:56.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:56 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3238371099' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abd76ef6-9f8b-4265-8fb7-b357fd8d1e28"}]': finished 2024-02-10T05:41:56.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:56 smithi110 ceph-mon[30869]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:41:56.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:56 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:41:57.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:56 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3238371099' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abd76ef6-9f8b-4265-8fb7-b357fd8d1e28"}]: dispatch 2024-02-10T05:41:57.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:56 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3238371099' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abd76ef6-9f8b-4265-8fb7-b357fd8d1e28"}]': finished 2024-02-10T05:41:57.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:56 smithi029 ceph-mon[25692]: osdmap e5: 1 total, 0 up, 1 in 2024-02-10T05:41:57.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:56 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:41:57.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:57 smithi105 ceph-mon[29854]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:57.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:57 smithi105 ceph-mon[29854]: Standby manager daemon x started 2024-02-10T05:41:57.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:57 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:41:57.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:57 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:41:57.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:57 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:41:57.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:57 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:41:57.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:57 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/948925694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:41:57.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:57 smithi110 ceph-mon[30869]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:57.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:57 smithi110 ceph-mon[30869]: Standby manager daemon x started 2024-02-10T05:41:57.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:57 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:41:57.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:57 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:41:57.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:57 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:41:57.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:57 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:41:57.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:57 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/948925694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:41:58.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:57 smithi029 ceph-mon[25692]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:41:58.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:57 smithi029 ceph-mon[25692]: Standby manager daemon x started 2024-02-10T05:41:58.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:57 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:41:58.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:57 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:41:58.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:57 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:41:58.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:57 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/319019752' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:41:58.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:57 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/948925694' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:41:58.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:58 smithi110 ceph-mon[30869]: mgrmap e15: y(active, since 83s), standbys: x, z 2024-02-10T05:41:58.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:58 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:41:59.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:58 smithi029 ceph-mon[25692]: mgrmap e15: y(active, since 83s), standbys: x, z 2024-02-10T05:41:59.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:58 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:41:59.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:58 smithi105 ceph-mon[29854]: mgrmap e15: y(active, since 83s), standbys: x, z 2024-02-10T05:41:59.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:58 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:41:59.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:41:59 smithi110 ceph-mon[30869]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:00.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:41:59 smithi029 ceph-mon[25692]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:00.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:41:59 smithi105 ceph-mon[29854]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:02.068 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:01 smithi029 ceph-mon[25692]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:02.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:01 smithi105 ceph-mon[29854]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:02.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:01 smithi110 ceph-mon[30869]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:03.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:02 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:42:03.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:02 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:03.140 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:02 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:42:03.141 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:02 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:03.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:02 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-10T05:42:03.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:02 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:04.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:03 smithi029 ceph-mon[25692]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:04.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:03 smithi029 ceph-mon[25692]: Deploying daemon osd.0 on smithi029 2024-02-10T05:42:04.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:03 smithi105 ceph-mon[29854]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:04.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:03 smithi105 ceph-mon[29854]: Deploying daemon osd.0 on smithi029 2024-02-10T05:42:04.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:03 smithi110 ceph-mon[30869]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:04.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:03 smithi110 ceph-mon[30869]: Deploying daemon osd.0 on smithi029 2024-02-10T05:42:06.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:05 smithi029 ceph-mon[25692]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:06.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:05 smithi105 ceph-mon[29854]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:06.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:05 smithi110 ceph-mon[30869]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:07.908 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:07 smithi029 ceph-mon[25692]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:08.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:07 smithi105 ceph-mon[29854]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:08.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:07 smithi110 ceph-mon[30869]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:09.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:09 smithi029 ceph-mon[25692]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:09.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:09 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:09.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:09 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:09.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:09 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:09.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:09 smithi105 ceph-mon[29854]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:09.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:09 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:09.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:09 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:09.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:09 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:09.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:09 smithi110 ceph-mon[30869]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:09.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:09 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:09.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:09 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:09.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:09 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:11.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:11 smithi029 ceph-mon[25692]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:11.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:11.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:11.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:11.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:11.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:11.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:11 smithi029 ceph-mon[25692]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:42:12.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:11 smithi105 ceph-mon[29854]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:11 smithi105 ceph-mon[29854]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:42:12.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:11 smithi110 ceph-mon[30869]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:12.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:12.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:12.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:12.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:12.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:12.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:11 smithi110 ceph-mon[30869]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-10T05:42:12.445 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-02-10T05:42:13.078 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.0.service 2024-02-10T05:42:13.081 INFO:tasks.cephadm:Deploying osd.1 on smithi105 with /dev/nvme4n1... 2024-02-10T05:42:13.081 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-10T05:42:13.324 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:42:13.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:13.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:42:13.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:42:13.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T05:42:13.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:13.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:13.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:13.375 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:13.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:13.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:42:13.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:42:13.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T05:42:13.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:13.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:13.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:13.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:13 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:13.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0[40338]: 2024-02-10T05:42:13.540+0000 7f222df81700 -1 osd.0 0 waiting for initial osdmap 2024-02-10T05:42:13.626 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:42:13 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0[40338]: 2024-02-10T05:42:13.543+0000 7f2225d6c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: osdmap e6: 1 total, 0 up, 1 in 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:13.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:13 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:14.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:14 smithi105 ceph-mon[29854]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T05:42:14.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:14 smithi105 ceph-mon[29854]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:42:14.279 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:14 smithi105 ceph-mon[29854]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' 2024-02-10T05:42:14.280 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.517 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:14 smithi029 ceph-mon[25692]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T05:42:14.517 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:14 smithi029 ceph-mon[25692]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:42:14.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:14 smithi029 ceph-mon[25692]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' 2024-02-10T05:42:14.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:14 smithi110 ceph-mon[30869]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-02-10T05:42:14.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:14 smithi110 ceph-mon[30869]: osdmap e7: 1 total, 0 up, 1 in 2024-02-10T05:42:14.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:14.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:14 smithi110 ceph-mon[30869]: from='osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305]' entity='osd.0' 2024-02-10T05:42:14.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:15.121 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/nvme4n1 2024-02-10T05:42:15.121 INFO:teuthology.orchestra.run.smithi105.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:42:15.121 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-10T05:42:15.121 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-02-10T05:42:15.121 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-02-10T05:42:15.121 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123316 s, 850 MB/s 2024-02-10T05:42:15.121 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-02-10T05:42:15.719 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:15 smithi105 ceph-mon[29854]: purged_snaps scrub starts 2024-02-10T05:42:15.719 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:15 smithi105 ceph-mon[29854]: purged_snaps scrub ok 2024-02-10T05:42:15.719 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:15 smithi105 ceph-mon[29854]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:15.719 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:15 smithi105 ceph-mon[29854]: osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305] boot 2024-02-10T05:42:15.720 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:15 smithi105 ceph-mon[29854]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:42:15.720 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:15 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:15.721 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi105:/dev/nvme4n1 2024-02-10T05:42:15.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:15 smithi029 ceph-mon[25692]: purged_snaps scrub starts 2024-02-10T05:42:15.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:15 smithi029 ceph-mon[25692]: purged_snaps scrub ok 2024-02-10T05:42:15.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:15 smithi029 ceph-mon[25692]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:15.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:15 smithi029 ceph-mon[25692]: osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305] boot 2024-02-10T05:42:15.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:15 smithi029 ceph-mon[25692]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:42:15.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:15 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:15.950 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:42:15.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:15 smithi110 ceph-mon[30869]: purged_snaps scrub starts 2024-02-10T05:42:15.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:15 smithi110 ceph-mon[30869]: purged_snaps scrub ok 2024-02-10T05:42:15.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:15 smithi110 ceph-mon[30869]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-10T05:42:15.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:15 smithi110 ceph-mon[30869]: osd.0 [v2:172.21.15.29:6802/1270944305,v1:172.21.15.29:6803/1270944305] boot 2024-02-10T05:42:15.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:15 smithi110 ceph-mon[30869]: osdmap e8: 1 total, 1 up, 1 in 2024-02-10T05:42:15.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:15 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:42:16.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:42:16.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:42:16.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:16.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:16.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:42:16.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:42:16.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:16.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:16.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: osdmap e9: 1 total, 1 up, 1 in 2024-02-10T05:42:16.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:42:16.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:16.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:16.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:16.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:17.654 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:17 smithi105 ceph-mon[29854]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:17.654 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:17 smithi105 ceph-mon[29854]: Detected new or changed devices on smithi029 2024-02-10T05:42:17.654 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:17 smithi105 ceph-mon[29854]: Adjusting osd_memory_target on smithi029 to 17093M 2024-02-10T05:42:17.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:17 smithi029 ceph-mon[25692]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:17.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:17 smithi029 ceph-mon[25692]: Detected new or changed devices on smithi029 2024-02-10T05:42:17.887 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:17 smithi029 ceph-mon[25692]: Adjusting osd_memory_target on smithi029 to 17093M 2024-02-10T05:42:17.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:17 smithi110 ceph-mon[30869]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:17.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:17 smithi110 ceph-mon[30869]: Detected new or changed devices on smithi029 2024-02-10T05:42:17.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:17 smithi110 ceph-mon[30869]: Adjusting osd_memory_target on smithi029 to 17093M 2024-02-10T05:42:18.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:18 smithi105 ceph-mon[29854]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:18.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:18 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:42:18.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:18 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:42:18.841 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:18 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:18.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:18 smithi029 ceph-mon[25692]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:18.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:18 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:42:18.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:18 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:42:18.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:18 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:18.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:18 smithi110 ceph-mon[30869]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:18.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:18 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:42:18.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:18 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:42:18.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:18 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:19.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:19 smithi105 ceph-mon[29854]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:19.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:19 smithi029 ceph-mon[25692]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:19.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:19 smithi110 ceph-mon[30869]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:20 smithi029 ceph-mon[25692]: from='client.? 172.21.15.105:0/4030292582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]: dispatch 2024-02-10T05:42:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:20 smithi029 ceph-mon[25692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]: dispatch 2024-02-10T05:42:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:20 smithi029 ceph-mon[25692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]': finished 2024-02-10T05:42:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:20 smithi029 ceph-mon[25692]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:42:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:20 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:20 smithi105 ceph-mon[29854]: from='client.? 172.21.15.105:0/4030292582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]: dispatch 2024-02-10T05:42:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:20 smithi105 ceph-mon[29854]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]: dispatch 2024-02-10T05:42:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:20 smithi105 ceph-mon[29854]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]': finished 2024-02-10T05:42:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:20 smithi105 ceph-mon[29854]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:42:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:20 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:21.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:20 smithi110 ceph-mon[30869]: from='client.? 172.21.15.105:0/4030292582' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]: dispatch 2024-02-10T05:42:21.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:20 smithi110 ceph-mon[30869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]: dispatch 2024-02-10T05:42:21.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:20 smithi110 ceph-mon[30869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9f8e40b1-b3fc-4aa0-a997-2fcb04c44498"}]': finished 2024-02-10T05:42:21.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:20 smithi110 ceph-mon[30869]: osdmap e10: 2 total, 1 up, 2 in 2024-02-10T05:42:21.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:20 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:21 smithi029 ceph-mon[25692]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:21 smithi029 ceph-mon[25692]: from='client.? 172.21.15.105:0/3284457715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:42:22.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:21 smithi105 ceph-mon[29854]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:22.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:21 smithi105 ceph-mon[29854]: from='client.? 172.21.15.105:0/3284457715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:42:22.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:21 smithi110 ceph-mon[30869]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:22.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:21 smithi110 ceph-mon[30869]: from='client.? 172.21.15.105:0/3284457715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:42:23.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:23 smithi105 ceph-mon[29854]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:24.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:23 smithi029 ceph-mon[25692]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:24.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:23 smithi110 ceph-mon[30869]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:25.827 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:25 smithi105 ceph-mon[29854]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:26.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:25 smithi029 ceph-mon[25692]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:26.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:25 smithi110 ceph-mon[30869]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:27.895 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:27 smithi105 ceph-mon[29854]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:27.895 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:27 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:42:27.895 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:27 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:27.895 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:27 smithi105 ceph-mon[29854]: Deploying daemon osd.1 on smithi105 2024-02-10T05:42:28.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:27 smithi029 ceph-mon[25692]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:28.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:27 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:42:28.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:27 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:28.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:27 smithi029 ceph-mon[25692]: Deploying daemon osd.1 on smithi105 2024-02-10T05:42:28.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:27 smithi110 ceph-mon[30869]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:28.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:27 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-10T05:42:28.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:27 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:28.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:27 smithi110 ceph-mon[30869]: Deploying daemon osd.1 on smithi105 2024-02-10T05:42:30.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:29 smithi029 ceph-mon[25692]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:30.141 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:29 smithi105 ceph-mon[29854]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:30.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:29 smithi110 ceph-mon[30869]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:31.904 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:31 smithi105 ceph-mon[29854]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:32.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:31 smithi029 ceph-mon[25692]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:32.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:31 smithi110 ceph-mon[30869]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:33.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:33 smithi110 ceph-mon[30869]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:33.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:33 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:33.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:33 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:33.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:33 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:33.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:33 smithi029 ceph-mon[25692]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:33.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:33 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:33.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:33 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:33.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:33 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:33.887 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:33 smithi105 ceph-mon[29854]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:33.887 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:33 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:33.887 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:33 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:33.887 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:33 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:35.823 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 1 on host 'smithi105' 2024-02-10T05:42:36.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.401 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:36.429 DEBUG:teuthology.orchestra.run.smithi105:osd.1> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.1.service 2024-02-10T05:42:36.431 INFO:tasks.cephadm:Deploying osd.2 on smithi110 with /dev/nvme4n1... 2024-02-10T05:42:36.431 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/nvme4n1 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:42:36.461 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-10T05:42:36.462 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.462 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:36.462 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:36.672 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:42:37.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:37 smithi105 ceph-mon[29854]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:42:37.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:37 smithi105 ceph-mon[29854]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:42:37.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:37.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:37 smithi105 ceph-mon[29854]: from='osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-02-10T05:42:37.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:37 smithi105 ceph-mon[29854]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-02-10T05:42:37.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:37 smithi110 ceph-mon[30869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:42:37.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:37 smithi110 ceph-mon[30869]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:42:37.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:37 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:37.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:37 smithi110 ceph-mon[30869]: from='osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-02-10T05:42:37.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:37 smithi110 ceph-mon[30869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-02-10T05:42:37.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:37 smithi029 ceph-mon[25692]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-10T05:42:37.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:37 smithi029 ceph-mon[25692]: osdmap e11: 2 total, 1 up, 2 in 2024-02-10T05:42:37.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:37 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:37.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:37 smithi029 ceph-mon[25692]: from='osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-02-10T05:42:37.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:37 smithi029 ceph-mon[25692]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-02-10T05:42:37.832 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:42:37 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1[34474]: 2024-02-10T05:42:37.596+0000 7f7f27b60700 -1 osd.1 0 waiting for initial osdmap 2024-02-10T05:42:37.832 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:42:37 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1[34474]: 2024-02-10T05:42:37.599+0000 7f7f1f94b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:42:38.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:38 smithi105 ceph-mon[29854]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:38.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:38 smithi105 ceph-mon[29854]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-02-10T05:42:38.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:38 smithi105 ceph-mon[29854]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:42:38.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:38.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:38 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:38.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:38 smithi105 ceph-mon[29854]: from='osd.1 ' entity='osd.1' 2024-02-10T05:42:38.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:38 smithi110 ceph-mon[30869]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:38.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:38 smithi110 ceph-mon[30869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-02-10T05:42:38.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:38 smithi110 ceph-mon[30869]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:42:38.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:38.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:38 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:38.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:38 smithi110 ceph-mon[30869]: from='osd.1 ' entity='osd.1' 2024-02-10T05:42:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping: /dev/nvme4n1 2024-02-10T05:42:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:--> --destroy was not specified, but zapping a whole device will remove the partition table 2024-02-10T05:42:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/nvme4n1 bs=1M count=10 conv=fsync 2024-02-10T05:42:38.549 INFO:teuthology.orchestra.run.smithi110.stderr: stderr: 10+0 records in 2024-02-10T05:42:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:10+0 records out 2024-02-10T05:42:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00960399 s, 1.1 GB/s 2024-02-10T05:42:38.549 INFO:teuthology.orchestra.run.smithi110.stderr:--> Zapping successful for: 2024-02-10T05:42:38.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:38 smithi029 ceph-mon[25692]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:38.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:38 smithi029 ceph-mon[25692]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-02-10T05:42:38.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:38 smithi029 ceph-mon[25692]: osdmap e12: 2 total, 1 up, 2 in 2024-02-10T05:42:38.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:38.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:38 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:38.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:38 smithi029 ceph-mon[25692]: from='osd.1 ' entity='osd.1' 2024-02-10T05:42:39.193 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi110:/dev/nvme4n1 2024-02-10T05:42:39.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:39 smithi105 ceph-mon[29854]: purged_snaps scrub starts 2024-02-10T05:42:39.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:39 smithi105 ceph-mon[29854]: purged_snaps scrub ok 2024-02-10T05:42:39.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:39 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:39.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:39 smithi105 ceph-mon[29854]: osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949] boot 2024-02-10T05:42:39.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:39 smithi105 ceph-mon[29854]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:42:39.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:39 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:39.466 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:42:39.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:39 smithi110 ceph-mon[30869]: purged_snaps scrub starts 2024-02-10T05:42:39.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:39 smithi110 ceph-mon[30869]: purged_snaps scrub ok 2024-02-10T05:42:39.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:39 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:39.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:39 smithi110 ceph-mon[30869]: osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949] boot 2024-02-10T05:42:39.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:39 smithi110 ceph-mon[30869]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:42:39.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:39 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:39.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:39 smithi029 ceph-mon[25692]: purged_snaps scrub starts 2024-02-10T05:42:39.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:39 smithi029 ceph-mon[25692]: purged_snaps scrub ok 2024-02-10T05:42:39.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:39 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:39.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:39 smithi029 ceph-mon[25692]: osd.1 [v2:172.21.15.105:6800/750351949,v1:172.21.15.105:6801/750351949] boot 2024-02-10T05:42:39.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:39 smithi029 ceph-mon[25692]: osdmap e13: 2 total, 2 up, 2 in 2024-02-10T05:42:39.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:39 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:40.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:40 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:40.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:40 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-02-10T05:42:40.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:42:40.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:40.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:40.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:40.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:40 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:41.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:41 smithi110 ceph-mon[30869]: Detected new or changed devices on smithi105 2024-02-10T05:42:41.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:41 smithi110 ceph-mon[30869]: Adjusting osd_memory_target on smithi105 to 17088M 2024-02-10T05:42:41.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:41 smithi110 ceph-mon[30869]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:41.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:41 smithi110 ceph-mon[30869]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:42:41.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:41 smithi029 ceph-mon[25692]: Detected new or changed devices on smithi105 2024-02-10T05:42:41.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:41 smithi029 ceph-mon[25692]: Adjusting osd_memory_target on smithi105 to 17088M 2024-02-10T05:42:41.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:41 smithi029 ceph-mon[25692]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:41.536 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:41 smithi029 ceph-mon[25692]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:42:41.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:41 smithi105 ceph-mon[29854]: Detected new or changed devices on smithi105 2024-02-10T05:42:41.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:41 smithi105 ceph-mon[29854]: Adjusting osd_memory_target on smithi105 to 17088M 2024-02-10T05:42:41.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:41 smithi105 ceph-mon[29854]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:41.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:41 smithi105 ceph-mon[29854]: osdmap e14: 2 total, 2 up, 2 in 2024-02-10T05:42:42.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:42 smithi110 ceph-mon[30869]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:42.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:42 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:42:42.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:42 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:42:42.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:42 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:42.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:42 smithi029 ceph-mon[25692]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:42.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:42 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:42:42.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:42 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:42:42.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:42 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:42.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:42 smithi105 ceph-mon[29854]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi110:/dev/nvme4n1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:42:42.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:42 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-10T05:42:42.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:42 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-10T05:42:42.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:42 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:43.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:43 smithi110 ceph-mon[30869]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:43.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:43 smithi029 ceph-mon[25692]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:43.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:43 smithi105 ceph-mon[29854]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:44.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:44 smithi110 ceph-mon[30869]: from='client.? 172.21.15.110:0/1357142779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]: dispatch 2024-02-10T05:42:44.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:44 smithi110 ceph-mon[30869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]: dispatch 2024-02-10T05:42:44.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:44 smithi110 ceph-mon[30869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]': finished 2024-02-10T05:42:44.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:44 smithi110 ceph-mon[30869]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:42:44.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:44 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:42:44.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:44 smithi029 ceph-mon[25692]: from='client.? 172.21.15.110:0/1357142779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]: dispatch 2024-02-10T05:42:44.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:44 smithi029 ceph-mon[25692]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]: dispatch 2024-02-10T05:42:44.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:44 smithi029 ceph-mon[25692]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]': finished 2024-02-10T05:42:44.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:44 smithi029 ceph-mon[25692]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:42:44.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:44 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:42:44.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:44 smithi105 ceph-mon[29854]: from='client.? 172.21.15.110:0/1357142779' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]: dispatch 2024-02-10T05:42:44.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:44 smithi105 ceph-mon[29854]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]: dispatch 2024-02-10T05:42:44.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:44 smithi105 ceph-mon[29854]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91673c64-7397-4fc1-a9f3-429e393fc0c3"}]': finished 2024-02-10T05:42:44.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:44 smithi105 ceph-mon[29854]: osdmap e15: 3 total, 2 up, 3 in 2024-02-10T05:42:44.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:44 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:42:46.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:45 smithi110 ceph-mon[30869]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:46.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:45 smithi110 ceph-mon[30869]: from='client.? 172.21.15.110:0/2092407104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:42:46.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:45 smithi029 ceph-mon[25692]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:46.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:45 smithi029 ceph-mon[25692]: from='client.? 172.21.15.110:0/2092407104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:42:46.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:45 smithi105 ceph-mon[29854]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:46.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:45 smithi105 ceph-mon[29854]: from='client.? 172.21.15.110:0/2092407104' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-10T05:42:48.020 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:47 smithi110 ceph-mon[30869]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:48.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:47 smithi029 ceph-mon[25692]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:48.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:47 smithi105 ceph-mon[29854]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:49.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:49 smithi110 ceph-mon[30869]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:50.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:49 smithi029 ceph-mon[25692]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:50.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:49 smithi105 ceph-mon[29854]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:51.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:50 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:42:51.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:50 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:51.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:42:51.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:50 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:51.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:50 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-10T05:42:51.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:50 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:52.113 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:51 smithi110 ceph-mon[30869]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:52.113 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:51 smithi110 ceph-mon[30869]: Deploying daemon osd.2 on smithi110 2024-02-10T05:42:52.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:51 smithi029 ceph-mon[25692]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:52.205 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:51 smithi029 ceph-mon[25692]: Deploying daemon osd.2 on smithi110 2024-02-10T05:42:52.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:51 smithi105 ceph-mon[29854]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:52.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:51 smithi105 ceph-mon[29854]: Deploying daemon osd.2 on smithi110 2024-02-10T05:42:54.138 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:53 smithi110 ceph-mon[30869]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:54.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:53 smithi029 ceph-mon[25692]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:54.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:53 smithi105 ceph-mon[29854]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:55.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:55 smithi110 ceph-mon[30869]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:56.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:55 smithi029 ceph-mon[25692]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:56.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:55 smithi105 ceph-mon[29854]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:57.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:57 smithi105 ceph-mon[29854]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:57.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:57 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:57.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:57 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:57.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:57 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:57.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:57 smithi029 ceph-mon[25692]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:57.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:57 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:57.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:57 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:57.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:57 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:57.678 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:57 smithi110 ceph-mon[30869]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:57.678 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:57 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:57.679 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:57 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:57.679 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:57 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:42:59.706 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:59.706 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.706 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.706 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:59.706 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:59.707 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.707 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: from='osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:42:59.707 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:42:59 smithi110 ceph-mon[30869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:42:59.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:59.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:59.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:59.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: from='osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:42:59.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:42:59 smithi029 ceph-mon[25692]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:42:59.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:42:59.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:42:59.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:42:59.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:42:59.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: from='osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:42:59.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:42:59 smithi105 ceph-mon[29854]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-10T05:43:00.020 INFO:teuthology.orchestra.run.smithi110.stdout:Created osd(s) 2 on host 'smithi110' 2024-02-10T05:43:00.593 DEBUG:teuthology.orchestra.run.smithi110:osd.2> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.2.service 2024-02-10T05:43:00.595 INFO:tasks.cephadm:Waiting for 3 OSDs to come up... 2024-02-10T05:43:00.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-10T05:43:00.844 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:00.879 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:43:00.880 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: from='osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-02-10T05:43:00.880 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:43:00.880 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:00.880 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-02-10T05:43:00.880 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:00.880 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:00.880 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:43:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: from='osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-02-10T05:43:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:43:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-02-10T05:43:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:00.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:00.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:00 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:00.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-10T05:43:00.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: from='osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-02-10T05:43:00.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: osdmap e16: 3 total, 2 up, 3 in 2024-02-10T05:43:00.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:00.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]: dispatch 2024-02-10T05:43:00.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:00.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:00.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:00 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:01.239 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2[36372]: 2024-02-10T05:43:00.897+0000 7fcacdb16700 -1 osd.2 0 waiting for initial osdmap 2024-02-10T05:43:01.240 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:00 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2[36372]: 2024-02-10T05:43:00.899+0000 7fcac5100700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-10T05:43:01.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:01 smithi110 ceph-mon[30869]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:43:01.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:01 smithi110 ceph-mon[30869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-02-10T05:43:01.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:01 smithi110 ceph-mon[30869]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:43:01.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:01 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:01.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:01 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:01.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:01 smithi110 ceph-mon[30869]: from='osd.2 ' entity='osd.2' 2024-02-10T05:43:01.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:01 smithi029 ceph-mon[25692]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:43:01.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:01 smithi029 ceph-mon[25692]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-02-10T05:43:01.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:01 smithi029 ceph-mon[25692]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:43:01.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:01 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:01.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:01 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:01.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:01 smithi029 ceph-mon[25692]: from='osd.2 ' entity='osd.2' 2024-02-10T05:43:01.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:01 smithi105 ceph-mon[29854]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:43:01.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:01 smithi105 ceph-mon[29854]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi110", "root=default"]}]': finished 2024-02-10T05:43:01.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:01 smithi105 ceph-mon[29854]: osdmap e17: 3 total, 2 up, 3 in 2024-02-10T05:43:01.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:01 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:01.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:01 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:01.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:01 smithi105 ceph-mon[29854]: from='osd.2 ' entity='osd.2' 2024-02-10T05:43:02.713 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:43:02.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:02 smithi029 ceph-mon[25692]: purged_snaps scrub starts 2024-02-10T05:43:02.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:02 smithi029 ceph-mon[25692]: purged_snaps scrub ok 2024-02-10T05:43:02.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:02 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:02.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:02 smithi029 ceph-mon[25692]: osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925] boot 2024-02-10T05:43:02.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:02 smithi029 ceph-mon[25692]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:43:02.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:02 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:02.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:02 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:43:02.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:02 smithi105 ceph-mon[29854]: purged_snaps scrub starts 2024-02-10T05:43:02.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:02 smithi105 ceph-mon[29854]: purged_snaps scrub ok 2024-02-10T05:43:02.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:02 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:02.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:02 smithi105 ceph-mon[29854]: osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925] boot 2024-02-10T05:43:02.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:02 smithi105 ceph-mon[29854]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:43:02.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:02 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:02.910 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:02 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:43:02.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:02 smithi110 ceph-mon[30869]: purged_snaps scrub starts 2024-02-10T05:43:02.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:02 smithi110 ceph-mon[30869]: purged_snaps scrub ok 2024-02-10T05:43:02.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:02 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:02.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:02 smithi110 ceph-mon[30869]: osd.2 [v2:172.21.15.110:6800/368518925,v1:172.21.15.110:6801/368518925] boot 2024-02-10T05:43:02.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:02 smithi110 ceph-mon[30869]: osdmap e18: 3 total, 3 up, 3 in 2024-02-10T05:43:02.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:02 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:02.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:02 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:43:03.386 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":18,"num_osds":3,"num_up_osds":3,"osd_up_since":1707543781,"num_in_osds":3,"osd_in_since":1707543763,"num_remapped_pgs":0} 2024-02-10T05:43:03.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:43:03.631 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:03.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:03 smithi029 ceph-mon[25692]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:43:03.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:03 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3740357327' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:43:03.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:03 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:43:03.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:03 smithi029 ceph-mon[25692]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:43:03.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:03 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:43:03.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:03 smithi110 ceph-mon[30869]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:43:03.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:03 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3740357327' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:43:03.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:03 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:43:03.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:03 smithi110 ceph-mon[30869]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:43:03.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:03 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:43:03.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:03 smithi105 ceph-mon[29854]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-02-10T05:43:03.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:03 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3740357327' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-10T05:43:03.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:03 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-10T05:43:03.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:03 smithi105 ceph-mon[29854]: osdmap e19: 3 total, 3 up, 3 in 2024-02-10T05:43:03.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:03 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-10T05:43:04.518 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46508]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-10T05:43:04.518 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46508]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:04.518 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46508]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:04.518 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46508]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:04.804 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46519]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-10T05:43:04.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46519]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:04.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46519]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:04.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:43:04 smithi029 sudo[46519]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:05.036 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:04 smithi105 sudo[40140]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-10T05:43:05.036 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:04 smithi105 sudo[40140]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:05.036 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:04 smithi105 sudo[40140]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:05.036 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:04 smithi105 sudo[40140]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:05.036 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: Detected new or changed devices on smithi110 2024-02-10T05:43:05.036 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.036 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.036 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: Adjusting osd_memory_target on smithi110 to 17088M 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:43:05.037 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:04 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: Detected new or changed devices on smithi110 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: Adjusting osd_memory_target on smithi110 to 17088M 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:43:05.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:43:05.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:04 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: Detected new or changed devices on smithi110 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: Adjusting osd_memory_target on smithi110 to 17088M 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: osdmap e20: 3 total, 3 up, 3 in 2024-02-10T05:43:05.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:04 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:05.409 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:05 smithi105 sudo[40143]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-10T05:43:05.409 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:05 smithi105 sudo[40143]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:05.409 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:05 smithi105 sudo[40143]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:05.409 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:43:05 smithi105 sudo[40143]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:05.557 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41184]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme4n1 2024-02-10T05:43:05.557 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41184]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:05.557 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41184]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:05.557 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41184]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:05.571 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:43:05.571 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"fsid":"65498394-c7d6-11ee-95b8-87774f69a715","created":"2024-02-10T05:39:37.775800+0000","modified":"2024-02-10T05:43:04.907487+0000","last_up_change":"2024-02-10T05:43:01.900021+0000","last_in_change":"2024-02-10T05:42:43.923486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:43:02.486509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abd76ef6-9f8b-4265-8fb7-b357fd8d1e28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6803","nonce":1270944305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6805","nonce":1270944305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6809","nonce":1270944305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6807","nonce":1270944305}]},"public_addr":"172.21.15.29:6803/1270944305","cluster_addr":"172.21.15.29:6805/1270944305","heartbeat_back_addr":"172.21.15.29:6809/1270944305","heartbeat_front_addr":"172.21.15.29:6807/1270944305","state":["exists","up"]},{"osd":1,"uuid":"9f8e40b1-b3fc-4aa0-a997-2fcb04c44498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6801","nonce":750351949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6803","nonce":750351949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6807","nonce":750351949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6805","nonce":750351949}]},"public_addr":"172.21.15.105:6801/750351949","cluster_addr":"172.21.15.105:6803/750351949","heartbeat_back_addr":"172.21.15.105:6807/750351949","heartbeat_front_addr":"172.21.15.105:6805/750351949","state":["exists","up"]},{"osd":2,"uuid":"91673c64-7397-4fc1-a9f3-429e393fc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6801","nonce":368518925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6803","nonce":368518925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6807","nonce":368518925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6805","nonce":368518925}]},"public_addr":"172.21.15.110:6801/368518925","cluster_addr":"172.21.15.110:6803/368518925","heartbeat_back_addr":"172.21.15.110:6807/368518925","heartbeat_front_addr":"172.21.15.110:6805/368518925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:42:12.530629+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:42:36.546264+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:43:00.375991+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/145104895":"2024-02-11T05:40:34.140567+0000","172.21.15.29:0/1388621034":"2024-02-11T05:40:34.140567+0000","172.21.15.29:6801/1467520796":"2024-02-11T05:40:34.140567+0000","172.21.15.29:0/333097797":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/224136733":"2024-02-11T05:40:07.795961+0000","172.21.15.29:6800/3786001490":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/2965721286":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/4050718709":"2024-02-11T05:40:34.140567+0000","172.21.15.29:6801/3786001490":"2024-02-11T05:40:07.795961+0000","172.21.15.29:6800/1467520796":"2024-02-11T05:40:34.140567+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:43:05.989 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41188]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme linux smart-log-add --json /dev/nvme4n1 2024-02-10T05:43:05.990 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41188]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:05.990 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41188]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:05.990 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:43:05 smithi110 sudo[41188]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:06.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:05 smithi029 sudo[46610]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:43:06.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:05 smithi029 sudo[46610]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:06.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:05 smithi029 sudo[46610]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:06.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:05 smithi029 sudo[46610]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:06.169 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-10T05:43:02.486509+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 1, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-10T05:43:06.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-10T05:43:06.382 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 ceph-mon[30869]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:06.382 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 ceph-mon[30869]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:43:06.382 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:06.382 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2415633540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:06.382 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:06.382 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 ceph-mon[30869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 sudo[46613]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 sudo[46613]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 sudo[46613]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 sudo[46613]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 ceph-mon[25692]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 ceph-mon[25692]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2415633540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:06.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:06.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:06 smithi029 ceph-mon[25692]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:06.407 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:06.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 ceph-mon[29854]: pgmap v85: 1 pgs: 1 active+clean; 449 KiB data, 88 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:06.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 ceph-mon[29854]: osdmap e21: 3 total, 3 up, 3 in 2024-02-10T05:43:06.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:06.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2415633540' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:06.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:06.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 ceph-mon[29854]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:06.634 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41201]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:43:06.634 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41201]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:06.634 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41201]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:06.634 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41201]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:06.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41205]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:43:06.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41205]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:06.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41205]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:06.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:06 smithi110 sudo[41205]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:07.150 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 sudo[40146]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-10T05:43:07.150 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 sudo[40146]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:07.150 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 sudo[40146]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:07.150 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:06 smithi105 sudo[40146]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:07.150 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 sudo[40149]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-10T05:43:07.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 sudo[40149]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 sudo[40149]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 sudo[40149]: pam_unix(sudo:session): session closed for user root 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: mgrmap e16: y(active, since 2m), standbys: x, z 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:07.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:07.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.411 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:07 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: mgrmap e16: y(active, since 2m), standbys: x, z 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:07.491 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:07.491 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.491 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.491 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:07 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.491 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:07.491 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: mgrmap e16: y(active, since 2m), standbys: x, z 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:07.492 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:07 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:08.237 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-02-10T05:43:08.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:08 smithi105 ceph-mon[29854]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:08.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:08 smithi105 ceph-mon[29854]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:08.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:08 smithi105 ceph-mon[29854]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:08.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:08 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:08.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:08 smithi110 ceph-mon[30869]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:08.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:08 smithi110 ceph-mon[30869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:08.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:08 smithi110 ceph-mon[30869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:08.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:08 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:08.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:08 smithi029 ceph-mon[25692]: pgmap v87: 1 pgs: 1 active+clean; 449 KiB data, 89 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:08.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:08 smithi029 ceph-mon[25692]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-10T05:43:08.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:08 smithi029 ceph-mon[25692]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-10T05:43:08.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:08 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:08.888 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi029 2024-02-10T05:43:08.888 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi029=foo.a' 2024-02-10T05:43:09.134 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:43:09.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:09 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3407477959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:43:09.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:09 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3407477959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:43:09.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:09 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3407477959' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-10T05:43:10.416 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:10 smithi110 ceph-mon[30869]: pgmap v88: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:10.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:10 smithi029 ceph-mon[25692]: pgmap v88: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:10.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:10 smithi105 ceph-mon[29854]: pgmap v88: 1 pgs: 1 active+clean; 449 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:10.813 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled rgw.foo.a update... 2024-02-10T05:43:11.377 DEBUG:teuthology.orchestra.run.smithi029:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@rgw.foo.a.service 2024-02-10T05:43:11.381 INFO:tasks.cephadm:Adding prometheus.a on smithi105 2024-02-10T05:43:11.381 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch apply prometheus '1;smithi105=a' 2024-02-10T05:43:11.615 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:43:11.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-02-10T05:43:11.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi029=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:11.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-02-10T05:43:11.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:11.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:11.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:11.991 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:11.991 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:11.991 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:43:11.991 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:43:11.991 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:11.991 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:11 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:12.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi029=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:43:12.128 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:12.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:11 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:12.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 10 KiB/s rd, 566 KiB/s wr, 36 op/s 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi029=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:12.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:11 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:12.948 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:12 smithi110 ceph-mon[30869]: Deploying daemon rgw.foo.a on smithi029 2024-02-10T05:43:13.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:12 smithi029 ceph-mon[25692]: Deploying daemon rgw.foo.a on smithi029 2024-02-10T05:43:13.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:12 smithi105 ceph-mon[29854]: Deploying daemon rgw.foo.a on smithi029 2024-02-10T05:43:13.365 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled prometheus update... 2024-02-10T05:43:13.817 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:43:13 smithi029 systemd[1]: Starting Ceph rgw.foo.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:43:13.924 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:13 smithi110 ceph-mon[30869]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.9 KiB/s rd, 424 KiB/s wr, 27 op/s 2024-02-10T05:43:13.925 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:13 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:13.926 DEBUG:teuthology.orchestra.run.smithi105:prometheus.a> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@prometheus.a.service 2024-02-10T05:43:13.929 INFO:tasks.cephadm:Adding node-exporter.a on smithi029 2024-02-10T05:43:13.929 INFO:tasks.cephadm:Adding node-exporter.b on smithi105 2024-02-10T05:43:13.929 INFO:tasks.cephadm:Adding node-exporter.c on smithi110 2024-02-10T05:43:13.930 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch apply node-exporter '3;smithi029=a;smithi105=b;smithi110=c' 2024-02-10T05:43:14.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:13 smithi029 ceph-mon[25692]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.9 KiB/s rd, 424 KiB/s wr, 27 op/s 2024-02-10T05:43:14.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:13 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:14.135 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:43:13 smithi029 podman[47147]: 2024-02-10 05:43:13.948919229 +0000 UTC m=+0.111596207 container create 6b1c9238d08979e1a10aeda3af15434dee508cc6413ae947ea6bb1aa1c7dc1fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:43:14.135 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:43:13 smithi029 podman[47147]: 2024-02-10 05:43:13.855080992 +0000 UTC m=+0.017757934 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:43:14.158 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:43:14.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:13 smithi105 ceph-mon[29854]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.9 KiB/s rd, 424 KiB/s wr, 27 op/s 2024-02-10T05:43:14.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:13 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:14.634 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:43:14 smithi029 podman[47147]: 2024-02-10 05:43:14.179905748 +0000 UTC m=+0.342582683 container init 6b1c9238d08979e1a10aeda3af15434dee508cc6413ae947ea6bb1aa1c7dc1fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:43:14.634 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:43:14 smithi029 podman[47147]: 2024-02-10 05:43:14.185263777 +0000 UTC m=+0.347940718 container start 6b1c9238d08979e1a10aeda3af15434dee508cc6413ae947ea6bb1aa1c7dc1fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131) 2024-02-10T05:43:14.634 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:43:14 smithi029 bash[47147]: 6b1c9238d08979e1a10aeda3af15434dee508cc6413ae947ea6bb1aa1c7dc1fc 2024-02-10T05:43:14.634 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:43:14 smithi029 systemd[1]: Started Ceph rgw.foo.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:43:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi105=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: Saving service prometheus spec with placement smithi105=a;count:1 2024-02-10T05:43:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:14 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:15.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi105=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:15.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: Saving service prometheus spec with placement smithi105=a;count:1 2024-02-10T05:43:15.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:14 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi105=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: Saving service prometheus spec with placement smithi105=a;count:1 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:15.221 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:14 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:16.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:15 smithi029 ceph-mon[25692]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.3 KiB/s rd, 339 KiB/s wr, 21 op/s 2024-02-10T05:43:16.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:15 smithi029 ceph-mon[25692]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-02-10T05:43:16.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:15 smithi029 ceph-mon[25692]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:43:16.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:15 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3300714031' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:43:16.148 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled node-exporter update... 2024-02-10T05:43:16.148 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:15 smithi110 ceph-mon[30869]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.3 KiB/s rd, 339 KiB/s wr, 21 op/s 2024-02-10T05:43:16.148 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:15 smithi110 ceph-mon[30869]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-02-10T05:43:16.148 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:15 smithi110 ceph-mon[30869]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:43:16.149 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:15 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3300714031' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:43:16.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:15 smithi105 ceph-mon[29854]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 6.3 KiB/s rd, 339 KiB/s wr, 21 op/s 2024-02-10T05:43:16.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:15 smithi105 ceph-mon[29854]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-02-10T05:43:16.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:15 smithi105 ceph-mon[29854]: osdmap e22: 3 total, 3 up, 3 in 2024-02-10T05:43:16.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:15 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3300714031' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-02-10T05:43:16.769 DEBUG:teuthology.orchestra.run.smithi029:node-exporter.a> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.a.service 2024-02-10T05:43:16.772 DEBUG:teuthology.orchestra.run.smithi105:node-exporter.b> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.b.service 2024-02-10T05:43:16.774 DEBUG:teuthology.orchestra.run.smithi110:node-exporter.c> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.c.service 2024-02-10T05:43:16.776 INFO:tasks.cephadm:Adding alertmanager.a on smithi029 2024-02-10T05:43:16.776 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch apply alertmanager '1;smithi029=a' 2024-02-10T05:43:17.044 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:43:17.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3300714031' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:43:17.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi029=a;smithi105=b;smithi110=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: Saving service node-exporter spec with placement smithi029=a;smithi105=b;smithi110=c;count:3 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:17.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:16 smithi029 ceph-mon[25692]: Checking dashboard <-> RGW credentials 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3300714031' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi029=a;smithi105=b;smithi110=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: Saving service node-exporter spec with placement smithi029=a;smithi105=b;smithi110=c;count:3 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:17.153 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:17.154 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:16 smithi110 ceph-mon[30869]: Checking dashboard <-> RGW credentials 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3300714031' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: osdmap e23: 3 total, 3 up, 3 in 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: from='client.24226 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "3;smithi029=a;smithi105=b;smithi110=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: Saving service node-exporter spec with placement smithi029=a;smithi105=b;smithi110=c;count:3 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:17.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:17.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:17.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:16 smithi105 ceph-mon[29854]: Checking dashboard <-> RGW credentials 2024-02-10T05:43:18.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:17 smithi029 ceph-mon[25692]: pgmap v94: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.9 KiB/s rd, 424 KiB/s wr, 27 op/s 2024-02-10T05:43:18.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:17 smithi029 ceph-mon[25692]: osdmap e24: 3 total, 3 up, 3 in 2024-02-10T05:43:18.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:17 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:43:18.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:17 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:43:18.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:17 smithi105 ceph-mon[29854]: pgmap v94: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.9 KiB/s rd, 424 KiB/s wr, 27 op/s 2024-02-10T05:43:18.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:17 smithi105 ceph-mon[29854]: osdmap e24: 3 total, 3 up, 3 in 2024-02-10T05:43:18.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:17 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:43:18.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:17 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:43:18.217 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:17 smithi110 ceph-mon[30869]: pgmap v94: 33 pgs: 8 creating+peering, 1 active+clean, 24 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 7.9 KiB/s rd, 424 KiB/s wr, 27 op/s 2024-02-10T05:43:18.218 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:17 smithi110 ceph-mon[30869]: osdmap e24: 3 total, 3 up, 3 in 2024-02-10T05:43:18.218 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:17 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:43:18.218 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:17 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-02-10T05:43:18.742 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled alertmanager update... 2024-02-10T05:43:19.040 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:18 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:43:19.040 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:18 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:43:19.040 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:18 smithi110 ceph-mon[30869]: osdmap e25: 3 total, 3 up, 3 in 2024-02-10T05:43:19.040 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:18 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:19.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:18 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:43:19.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:18 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:43:19.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:18 smithi029 ceph-mon[25692]: osdmap e25: 3 total, 3 up, 3 in 2024-02-10T05:43:19.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:18 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:19.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:18 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:43:19.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:18 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-02-10T05:43:19.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:18 smithi105 ceph-mon[29854]: osdmap e25: 3 total, 3 up, 3 in 2024-02-10T05:43:19.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:18 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:19.333 DEBUG:teuthology.orchestra.run.smithi029:alertmanager.a> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@alertmanager.a.service 2024-02-10T05:43:19.335 INFO:tasks.cephadm:Adding grafana.a on smithi105 2024-02-10T05:43:19.336 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph orch apply grafana '1;smithi105=a' 2024-02-10T05:43:19.564 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:43:19.926 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:19 smithi110 ceph-mon[30869]: pgmap v97: 65 pgs: 17 creating+peering, 15 active+clean, 33 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T05:43:19.926 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:19 smithi110 ceph-mon[30869]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:19.926 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:19 smithi110 ceph-mon[30869]: Saving service alertmanager spec with placement smithi029=a;count:1 2024-02-10T05:43:19.926 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:19 smithi110 ceph-mon[30869]: osdmap e26: 3 total, 3 up, 3 in 2024-02-10T05:43:19.926 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:19 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:43:19.927 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:19 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:43:19.927 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:19 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:20.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:19 smithi029 ceph-mon[25692]: pgmap v97: 65 pgs: 17 creating+peering, 15 active+clean, 33 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T05:43:20.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:19 smithi029 ceph-mon[25692]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:20.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:19 smithi029 ceph-mon[25692]: Saving service alertmanager spec with placement smithi029=a;count:1 2024-02-10T05:43:20.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:19 smithi029 ceph-mon[25692]: osdmap e26: 3 total, 3 up, 3 in 2024-02-10T05:43:20.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:19 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:43:20.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:19 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:43:20.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:19 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:20.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:19 smithi105 ceph-mon[29854]: pgmap v97: 65 pgs: 17 creating+peering, 15 active+clean, 33 unknown; 577 KiB data, 91 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-02-10T05:43:20.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:19 smithi105 ceph-mon[29854]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:20.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:19 smithi105 ceph-mon[29854]: Saving service alertmanager spec with placement smithi029=a;count:1 2024-02-10T05:43:20.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:19 smithi105 ceph-mon[29854]: osdmap e26: 3 total, 3 up, 3 in 2024-02-10T05:43:20.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:19 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:43:20.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:19 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-02-10T05:43:20.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:19 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:20.949 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:20 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:43:20.949 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:20 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:43:20.949 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:20 smithi110 ceph-mon[30869]: osdmap e27: 3 total, 3 up, 3 in 2024-02-10T05:43:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:20 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:43:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:20 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:43:21.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:20 smithi029 ceph-mon[25692]: osdmap e27: 3 total, 3 up, 3 in 2024-02-10T05:43:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:20 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:43:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:20 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-02-10T05:43:21.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:20 smithi105 ceph-mon[29854]: osdmap e27: 3 total, 3 up, 3 in 2024-02-10T05:43:21.389 INFO:teuthology.orchestra.run.smithi110.stdout:Scheduled grafana update... 2024-02-10T05:43:21.910 DEBUG:teuthology.orchestra.run.smithi105:grafana.a> sudo journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@grafana.a.service 2024-02-10T05:43:21.913 INFO:tasks.cephadm:Setting up client nodes... 2024-02-10T05:43:21.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:43:21.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:21 smithi110 ceph-mon[30869]: pgmap v100: 97 pgs: 26 creating+peering, 36 active+clean, 35 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T05:43:21.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:21 smithi110 ceph-mon[30869]: osdmap e28: 3 total, 3 up, 3 in 2024-02-10T05:43:21.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:21 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:43:21.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:21 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:43:21.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:21 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:21 smithi029 ceph-mon[25692]: pgmap v100: 97 pgs: 26 creating+peering, 36 active+clean, 35 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T05:43:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:21 smithi029 ceph-mon[25692]: osdmap e28: 3 total, 3 up, 3 in 2024-02-10T05:43:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:21 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:43:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:21 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:43:22.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:21 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:22.157 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:22.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:21 smithi105 ceph-mon[29854]: pgmap v100: 97 pgs: 26 creating+peering, 36 active+clean, 35 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-02-10T05:43:22.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:21 smithi105 ceph-mon[29854]: osdmap e28: 3 total, 3 up, 3 in 2024-02-10T05:43:22.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:21 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:43:22.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:21 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-02-10T05:43:22.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:21 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:22 smithi029 ceph-mon[25692]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi105=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:22 smithi029 ceph-mon[25692]: Saving service grafana spec with placement smithi105=a;count:1 2024-02-10T05:43:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:22 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:43:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:22 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:43:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:22 smithi029 ceph-mon[25692]: osdmap e29: 3 total, 3 up, 3 in 2024-02-10T05:43:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:22 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:43:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:22 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:43:23.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:22 smithi105 ceph-mon[29854]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi105=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:23.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:22 smithi105 ceph-mon[29854]: Saving service grafana spec with placement smithi105=a;count:1 2024-02-10T05:43:23.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:22 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:43:23.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:22 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:43:23.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:22 smithi105 ceph-mon[29854]: osdmap e29: 3 total, 3 up, 3 in 2024-02-10T05:43:23.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:22 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:43:23.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:22 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:43:23.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:22 smithi110 ceph-mon[30869]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi105=a", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:43:23.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:22 smithi110 ceph-mon[30869]: Saving service grafana spec with placement smithi105=a;count:1 2024-02-10T05:43:23.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:22 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:43:23.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:22 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-02-10T05:43:23.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:22 smithi110 ceph-mon[30869]: osdmap e29: 3 total, 3 up, 3 in 2024-02-10T05:43:23.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:22 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:43:23.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:22 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-02-10T05:43:24.112 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-02-10T05:43:24.112 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQD8DMdlWpKTBhAALOFcc25Y7nwA1JLz3+tpQA== 2024-02-10T05:43:24.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:23 smithi029 ceph-mon[25692]: pgmap v103: 129 pgs: 18 creating+peering, 67 active+clean, 44 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-10T05:43:24.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:23 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:43:24.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:23 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:43:24.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:23 smithi029 ceph-mon[25692]: osdmap e30: 3 total, 3 up, 3 in 2024-02-10T05:43:24.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:23 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:23 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:23 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:23 smithi105 ceph-mon[29854]: pgmap v103: 129 pgs: 18 creating+peering, 67 active+clean, 44 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-10T05:43:24.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:23 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:43:24.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:23 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:43:24.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:23 smithi105 ceph-mon[29854]: osdmap e30: 3 total, 3 up, 3 in 2024-02-10T05:43:24.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:23 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:23 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:23 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:23 smithi110 ceph-mon[30869]: pgmap v103: 129 pgs: 18 creating+peering, 67 active+clean, 44 unknown; 578 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-02-10T05:43:24.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:23 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/693295343' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:43:24.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:23 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2045253004' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-02-10T05:43:24.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:23 smithi110 ceph-mon[30869]: osdmap e30: 3 total, 3 up, 3 in 2024-02-10T05:43:24.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:23 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:23 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:23 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:24.762 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:43:24.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-10T05:43:24.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-10T05:43:24.805 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:43:25.079 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.b/config 2024-02-10T05:43:25.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:24 smithi029 ceph-mon[25692]: Deploying daemon prometheus.a on smithi105 2024-02-10T05:43:25.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:24 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2457589508' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:25.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:24 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2457589508' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:25.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:24 smithi105 ceph-mon[29854]: Deploying daemon prometheus.a on smithi105 2024-02-10T05:43:25.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:24 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2457589508' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:25.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:24 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2457589508' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:25.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:24 smithi110 ceph-mon[30869]: Deploying daemon prometheus.a on smithi105 2024-02-10T05:43:25.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:24 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2457589508' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:25.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:24 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2457589508' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:26.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:26 smithi029 ceph-mon[25692]: pgmap v105: 129 pgs: 119 active+clean, 10 unknown; 579 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 50 KiB/s rd, 3.5 KiB/s wr, 116 op/s 2024-02-10T05:43:26.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:26 smithi105 ceph-mon[29854]: pgmap v105: 129 pgs: 119 active+clean, 10 unknown; 579 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 50 KiB/s rd, 3.5 KiB/s wr, 116 op/s 2024-02-10T05:43:26.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:26 smithi110 ceph-mon[30869]: pgmap v105: 129 pgs: 119 active+clean, 10 unknown; 579 KiB data, 94 MiB used, 268 GiB / 268 GiB avail; 50 KiB/s rd, 3.5 KiB/s wr, 116 op/s 2024-02-10T05:43:27.774 INFO:teuthology.orchestra.run.smithi105.stdout:[client.1] 2024-02-10T05:43:27.774 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQD/DMdlgePsLRAAm1AEeZDkhNHxhMGfFfrxRw== 2024-02-10T05:43:28.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:28 smithi029 ceph-mon[25692]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.3 KiB/s wr, 183 op/s 2024-02-10T05:43:28.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:28 smithi029 ceph-mon[25692]: from='client.? 172.21.15.105:0/4018426035' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:28.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:28 smithi029 ceph-mon[25692]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:28.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:28 smithi029 ceph-mon[25692]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:28.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:28 smithi105 ceph-mon[29854]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.3 KiB/s wr, 183 op/s 2024-02-10T05:43:28.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:28 smithi105 ceph-mon[29854]: from='client.? 172.21.15.105:0/4018426035' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:28.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:28 smithi105 ceph-mon[29854]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:28.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:28 smithi105 ceph-mon[29854]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:28.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:28 smithi110 ceph-mon[30869]: pgmap v106: 129 pgs: 129 active+clean; 581 KiB data, 98 MiB used, 268 GiB / 268 GiB avail; 81 KiB/s rd, 4.3 KiB/s wr, 183 op/s 2024-02-10T05:43:28.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:28 smithi110 ceph-mon[30869]: from='client.? 172.21.15.105:0/4018426035' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:28.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:28 smithi110 ceph-mon[30869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:28.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:28 smithi110 ceph-mon[30869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:30.096 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:43:30.097 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-02-10T05:43:30.097 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-02-10T05:43:30.141 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-10T05:43:30.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:30 smithi029 ceph-mon[25692]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 98 KiB/s rd, 4.8 KiB/s wr, 219 op/s 2024-02-10T05:43:30.384 INFO:teuthology.orchestra.run.smithi110.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.c/config 2024-02-10T05:43:30.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:30 smithi105 ceph-mon[29854]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 98 KiB/s rd, 4.8 KiB/s wr, 219 op/s 2024-02-10T05:43:30.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:30 smithi110 ceph-mon[30869]: pgmap v107: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 98 KiB/s rd, 4.8 KiB/s wr, 219 op/s 2024-02-10T05:43:32.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:32 smithi105 ceph-mon[29854]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 86 KiB/s rd, 4.2 KiB/s wr, 192 op/s 2024-02-10T05:43:32.188 INFO:teuthology.orchestra.run.smithi110.stdout:[client.2] 2024-02-10T05:43:32.188 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAEDcdlQDcECxAAPxovh/avnPCJsM/a0DKJAg== 2024-02-10T05:43:32.188 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:32 smithi110 ceph-mon[30869]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 86 KiB/s rd, 4.2 KiB/s wr, 192 op/s 2024-02-10T05:43:32.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:32 smithi029 ceph-mon[25692]: pgmap v108: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 86 KiB/s rd, 4.2 KiB/s wr, 192 op/s 2024-02-10T05:43:33.003 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:43:33.003 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-02-10T05:43:33.003 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-02-10T05:43:33.047 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-10T05:43:33.047 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-10T05:43:33.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-10T05:43:33.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:33 smithi110 ceph-mon[30869]: from='client.? 172.21.15.110:0/3924789626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:33.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:33 smithi110 ceph-mon[30869]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:33.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:33 smithi110 ceph-mon[30869]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:33.330 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:33 smithi105 ceph-mon[29854]: from='client.? 172.21.15.110:0/3924789626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:33.330 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:33 smithi105 ceph-mon[29854]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:33.331 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:33 smithi105 ceph-mon[29854]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:33.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:33 smithi029 ceph-mon[25692]: from='client.? 172.21.15.110:0/3924789626' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:33.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:33 smithi029 ceph-mon[25692]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-10T05:43:33.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:33 smithi029 ceph-mon[25692]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-10T05:43:34.323 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:34 smithi105 ceph-mon[29854]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 160 op/s 2024-02-10T05:43:34.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:34 smithi029 ceph-mon[25692]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 160 op/s 2024-02-10T05:43:34.485 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:34 smithi110 ceph-mon[30869]: pgmap v109: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 72 KiB/s rd, 3.5 KiB/s wr, 160 op/s 2024-02-10T05:43:34.660 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:34 smithi105 systemd[1]: Starting Ceph prometheus.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:43:35.159 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:34 smithi105 podman[41072]: 2024-02-10 05:43:34.620156722 +0000 UTC m=+0.018452110 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-02-10T05:43:35.206 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:43:35.452 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-mon[29854]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 64 KiB/s rd, 3.1 KiB/s wr, 141 op/s 2024-02-10T05:43:35.452 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:35.452 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2440264000' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:43:35.453 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 podman[41072]: 2024-02-10 05:43:35.186967295 +0000 UTC m=+0.585262679 container create 7741dbf83128b587b21021c12990c19028cdaf6eb0a4d4c1171c4286f9562d02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:43:35.453 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 podman[41072]: 2024-02-10 05:43:35.397276725 +0000 UTC m=+0.795572111 container init 7741dbf83128b587b21021c12990c19028cdaf6eb0a4d4c1171c4286f9562d02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:43:35.453 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 podman[41072]: 2024-02-10 05:43:35.403598393 +0000 UTC m=+0.801893778 container start 7741dbf83128b587b21021c12990c19028cdaf6eb0a4d4c1171c4286f9562d02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:43:35.453 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.429Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-02-10T05:43:35.453 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.429Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-02-10T05:43:35.453 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.429Z caller=main.go:570 level=info host_details="(Linux 5.14.0-419.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Feb 7 23:01:41 UTC 2024 x86_64 smithi105 (none))" 2024-02-10T05:43:35.453 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.429Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.429Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.431Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.431Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.432Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.432Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.434Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.434Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.491µs 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.434Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.434Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.435Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.829µs wal_replay_duration=296.73µs wbl_replay_duration=161ns total_replay_duration=413.256µs 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.436Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.436Z caller=main.go:1029 level=info msg="TSDB started" 2024-02-10T05:43:35.454 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.436Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-02-10T05:43:35.455 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.452Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.638786ms db_storage=957ns remote_storage=1.061µs web_handler=418ns query_engine=999ns scrape=425.28µs scrape_sd=72.147µs notify=1.146µs notify_sd=19.913µs rules=15.931364ms tracing=4.225µs 2024-02-10T05:43:35.455 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.452Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-02-10T05:43:35.455 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:43:35.452Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-02-10T05:43:35.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:35 smithi110 ceph-mon[30869]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 64 KiB/s rd, 3.1 KiB/s wr, 141 op/s 2024-02-10T05:43:35.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:35 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:35.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:35 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2440264000' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:43:35.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:35 smithi029 ceph-mon[25692]: pgmap v110: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 64 KiB/s rd, 3.1 KiB/s wr, 141 op/s 2024-02-10T05:43:35.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:35 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:35.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:35 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2440264000' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-10T05:43:35.908 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":647437633},{"type":"v1","addr":"172.21.15.29:6801","nonce":647437633}]},"active_addr":"172.21.15.29:6801/647437633","active_change":"2024-02-10T05:40:34.140678+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":14223,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3046643352}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":216693107}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":801316162}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1874671034}]}]} 2024-02-10T05:43:35.915 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-10T05:43:35.915 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-10T05:43:35.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:43:35.916 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 bash[41072]: 7741dbf83128b587b21021c12990c19028cdaf6eb0a4d4c1171c4286f9562d02 2024-02-10T05:43:35.916 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:43:35 smithi105 systemd[1]: Started Ceph prometheus.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:43:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:36.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.885 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:36 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:43:36.885 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:36 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ignoring --setuser ceph since I am not root 2024-02-10T05:43:36.885 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:36 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ignoring --setgroup ceph since I am not root 2024-02-10T05:43:36.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: ignoring --setuser ceph since I am not root 2024-02-10T05:43:36.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: ignoring --setgroup ceph since I am not root 2024-02-10T05:43:36.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:36 smithi105 ceph-mgr[31377]: -- 172.21.15.105:0/636225563 <== mon.2 v2:172.21.15.105:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a78feb81a0 con 0x55a78fe8ec00 2024-02-10T05:43:36.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.909 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:36 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:43:36.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' 2024-02-10T05:43:36.990 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:36 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-02-10T05:43:36.990 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:36 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: ignoring --setuser ceph since I am not root 2024-02-10T05:43:36.990 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:36 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: ignoring --setgroup ceph since I am not root 2024-02-10T05:43:36.990 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:36 smithi110 ceph-mgr[33213]: -- 172.21.15.110:0/2413655026 <== mon.1 v2:172.21.15.110:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564a2b4541a0 con 0x564a2b42ac00 2024-02-10T05:43:37.265 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:37.168+0000 7fd74e512200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:43:37.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:37 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:37.152+0000 7f51c8e33200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:43:37.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:37 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:37.246+0000 7f51c8e33200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:43:37.489 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:37 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:37.149+0000 7f7763781200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-10T05:43:37.489 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:37 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:37.242+0000 7f7763781200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:43:37.587 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:37.264+0000 7fd74e512200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-10T05:43:37.872 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:37 smithi110 ceph-mon[30869]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:43:37.872 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:37 smithi110 ceph-mon[30869]: mgrmap e17: y(active, since 3m), standbys: x, z 2024-02-10T05:43:37.880 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:37 smithi105 ceph-mon[29854]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:43:37.880 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:37 smithi105 ceph-mon[29854]: mgrmap e17: y(active, since 3m), standbys: x, z 2024-02-10T05:43:37.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:37 smithi029 ceph-mon[25692]: from='mgr.14152 172.21.15.29:0/1420686239' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-02-10T05:43:37.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:37 smithi029 ceph-mon[25692]: mgrmap e17: y(active, since 3m), standbys: x, z 2024-02-10T05:43:38.049 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:43:38.049 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"fsid":"65498394-c7d6-11ee-95b8-87774f69a715","created":"2024-02-10T05:39:37.775800+0000","modified":"2024-02-10T05:43:22.864359+0000","last_up_change":"2024-02-10T05:43:01.900021+0000","last_in_change":"2024-02-10T05:42:43.923486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:43:02.486509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-10T05:43:14.467340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-02-10T05:43:16.143658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-02-10T05:43:18.098843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-02-10T05:43:20.170293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abd76ef6-9f8b-4265-8fb7-b357fd8d1e28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6803","nonce":1270944305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6805","nonce":1270944305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6809","nonce":1270944305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6807","nonce":1270944305}]},"public_addr":"172.21.15.29:6803/1270944305","cluster_addr":"172.21.15.29:6805/1270944305","heartbeat_back_addr":"172.21.15.29:6809/1270944305","heartbeat_front_addr":"172.21.15.29:6807/1270944305","state":["exists","up"]},{"osd":1,"uuid":"9f8e40b1-b3fc-4aa0-a997-2fcb04c44498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6801","nonce":750351949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6803","nonce":750351949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6807","nonce":750351949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6805","nonce":750351949}]},"public_addr":"172.21.15.105:6801/750351949","cluster_addr":"172.21.15.105:6803/750351949","heartbeat_back_addr":"172.21.15.105:6807/750351949","heartbeat_front_addr":"172.21.15.105:6805/750351949","state":["exists","up"]},{"osd":2,"uuid":"91673c64-7397-4fc1-a9f3-429e393fc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6801","nonce":368518925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6803","nonce":368518925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6807","nonce":368518925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6805","nonce":368518925}]},"public_addr":"172.21.15.110:6801/368518925","cluster_addr":"172.21.15.110:6803/368518925","heartbeat_back_addr":"172.21.15.110:6807/368518925","heartbeat_front_addr":"172.21.15.110:6805/368518925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:42:12.530629+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:42:36.546264+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:43:00.375991+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/145104895":"2024-02-11T05:40:34.140567+0000","172.21.15.29:0/1388621034":"2024-02-11T05:40:34.140567+0000","172.21.15.29:6801/1467520796":"2024-02-11T05:40:34.140567+0000","172.21.15.29:0/333097797":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/224136733":"2024-02-11T05:40:07.795961+0000","172.21.15.29:6800/3786001490":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/2965721286":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/4050718709":"2024-02-11T05:40:34.140567+0000","172.21.15.29:6801/3786001490":"2024-02-11T05:40:07.795961+0000","172.21.15.29:6800/1467520796":"2024-02-11T05:40:34.140567+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:43:38.159 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:37 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:37.880+0000 7f51c8e33200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:43:38.159 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:37 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:37.972+0000 7f51c8e33200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:43:38.159 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:38.111+0000 7f51c8e33200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:43:38.197 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:37 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:37.871+0000 7f7763781200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:43:38.197 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:37 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:37.963+0000 7f7763781200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:43:38.197 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.102+0000 7f7763781200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:43:38.197 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.196+0000 7f7763781200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:43:38.292 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:37.945+0000 7fd74e512200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-10T05:43:38.292 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.041+0000 7fd74e512200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-10T05:43:38.292 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.189+0000 7fd74e512200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-10T05:43:38.292 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.291+0000 7fd74e512200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:43:38.489 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.278+0000 7f7763781200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:43:38.556 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:38.206+0000 7f51c8e33200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-10T05:43:38.556 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:38.286+0000 7f51c8e33200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:43:38.557 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:38.556+0000 7f51c8e33200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:43:38.588 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.378+0000 7fd74e512200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-10T05:43:38.804 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-10T05:43:38.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-10T05:43:38.825 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3955755107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:38.825 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.538+0000 7f7763781200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:43:38.825 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.646+0000 7f7763781200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:43:38.825 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.731+0000 7f7763781200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:43:38.826 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.825+0000 7f7763781200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:43:38.846 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3955755107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:38.846 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:38.666+0000 7f51c8e33200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:43:38.846 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:38.751+0000 7f51c8e33200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:43:38.846 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:38.845+0000 7f51c8e33200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:43:38.854 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3955755107' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:38.854 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.648+0000 7fd74e512200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-10T05:43:38.854 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.761+0000 7fd74e512200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-10T05:43:38.855 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.854+0000 7fd74e512200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-10T05:43:39.045 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:39.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:38 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:38.954+0000 7fd74e512200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-10T05:43:39.159 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:39 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:39.018+0000 7f51c8e33200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:43:39.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:38 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:38.998+0000 7f7763781200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:43:39.405 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:39 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:39.142+0000 7fd74e512200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-10T05:43:40.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:39 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:39.947+0000 7f7763781200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:43:40.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:40 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:40.076+0000 7f7763781200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:43:40.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:39 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:39.973+0000 7f51c8e33200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:43:40.409 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:40 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:40.102+0000 7f51c8e33200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:43:40.428 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:40 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:40.156+0000 7fd74e512200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-10T05:43:40.428 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:40 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:40.295+0000 7fd74e512200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-10T05:43:40.688 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:40 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:40.405+0000 7f7763781200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:43:40.688 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:40 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:40.494+0000 7f7763781200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:43:40.688 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:40 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:40.687+0000 7f7763781200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:43:40.716 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:40 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:40.431+0000 7f51c8e33200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:43:40.716 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:40 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:40.521+0000 7f51c8e33200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:43:40.716 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:40 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:40.715+0000 7f51c8e33200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:43:40.740 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:40 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:40.643+0000 7fd74e512200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-10T05:43:40.740 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:40 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:40.739+0000 7fd74e512200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-10T05:43:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:43:40.873 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"fsid":"65498394-c7d6-11ee-95b8-87774f69a715","created":"2024-02-10T05:39:37.775800+0000","modified":"2024-02-10T05:43:22.864359+0000","last_up_change":"2024-02-10T05:43:01.900021+0000","last_in_change":"2024-02-10T05:42:43.923486+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-10T05:43:02.486509+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":3,"score_stable":3,"optimal_score":1,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-02-10T05:43:14.467340+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"default.rgw.log","create_time":"2024-02-10T05:43:16.143658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.control","create_time":"2024-02-10T05:43:18.098843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.1299999952316284,"score_stable":1.1299999952316284,"optimal_score":1,"raw_score_acting":1.1299999952316284,"raw_score_stable":1.1299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.meta","create_time":"2024-02-10T05:43:20.170293+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.2200000286102295,"score_stable":1.2200000286102295,"optimal_score":1,"raw_score_acting":1.2200000286102295,"raw_score_stable":1.2200000286102295,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"abd76ef6-9f8b-4265-8fb7-b357fd8d1e28","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6803","nonce":1270944305}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6805","nonce":1270944305}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6809","nonce":1270944305}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":1270944305},{"type":"v1","addr":"172.21.15.29:6807","nonce":1270944305}]},"public_addr":"172.21.15.29:6803/1270944305","cluster_addr":"172.21.15.29:6805/1270944305","heartbeat_back_addr":"172.21.15.29:6809/1270944305","heartbeat_front_addr":"172.21.15.29:6807/1270944305","state":["exists","up"]},{"osd":1,"uuid":"9f8e40b1-b3fc-4aa0-a997-2fcb04c44498","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6801","nonce":750351949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6803","nonce":750351949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6807","nonce":750351949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":750351949},{"type":"v1","addr":"172.21.15.105:6805","nonce":750351949}]},"public_addr":"172.21.15.105:6801/750351949","cluster_addr":"172.21.15.105:6803/750351949","heartbeat_back_addr":"172.21.15.105:6807/750351949","heartbeat_front_addr":"172.21.15.105:6805/750351949","state":["exists","up"]},{"osd":2,"uuid":"91673c64-7397-4fc1-a9f3-429e393fc0c3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":28,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6801","nonce":368518925}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6803","nonce":368518925}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6807","nonce":368518925}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":368518925},{"type":"v1","addr":"172.21.15.110:6805","nonce":368518925}]},"public_addr":"172.21.15.110:6801/368518925","cluster_addr":"172.21.15.110:6803/368518925","heartbeat_back_addr":"172.21.15.110:6807/368518925","heartbeat_front_addr":"172.21.15.110:6805/368518925","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:42:12.530629+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:42:36.546264+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-10T05:43:00.375991+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/145104895":"2024-02-11T05:40:34.140567+0000","172.21.15.29:0/1388621034":"2024-02-11T05:40:34.140567+0000","172.21.15.29:6801/1467520796":"2024-02-11T05:40:34.140567+0000","172.21.15.29:0/333097797":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/224136733":"2024-02-11T05:40:07.795961+0000","172.21.15.29:6800/3786001490":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/2965721286":"2024-02-11T05:40:07.795961+0000","172.21.15.29:0/4050718709":"2024-02-11T05:40:34.140567+0000","172.21.15.29:6801/3786001490":"2024-02-11T05:40:07.795961+0000","172.21.15.29:6800/1467520796":"2024-02-11T05:40:34.140567+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-10T05:43:40.989 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:40 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/751089862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:40.989 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:40 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:40.868+0000 7f7763781200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:43:41.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:40 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/751089862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:41.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:40 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:40.948+0000 7fd74e512200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:43:41.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:40 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/751089862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-10T05:43:41.159 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:40 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:40.899+0000 7f51c8e33200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:43:41.401 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:41 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:41.147+0000 7f7763781200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:43:41.401 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:41 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:41.401+0000 7f7763781200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:43:41.436 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:41 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:41.140+0000 7fd74e512200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-10T05:43:41.438 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:41 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:41.180+0000 7f51c8e33200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:43:41.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-10T05:43:41.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-10T05:43:41.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-10T05:43:41.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:41 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:41.436+0000 7fd74e512200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-10T05:43:41.710 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:41 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:41.709+0000 7fd74e512200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:43:41.739 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:41 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:41.632+0000 7f7763781200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:43:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:41.820 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:41.821 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:41.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:41 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:41.437+0000 7f51c8e33200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-10T05:43:41.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:41 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:41.670+0000 7f51c8e33200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:43:42.266 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:42 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:42.012+0000 7fd74e512200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-10T05:43:42.739 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:42 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:42.417+0000 7f7763781200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:43:42.909 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:42 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:42.463+0000 7f51c8e33200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:43:43.041 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:42 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:42.885+0000 7fd74e512200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-10T05:43:43.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:42 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:42.978+0000 7f7763781200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:43:43.239 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:43.074+0000 7f7763781200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:43:43.333 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:43.028+0000 7f51c8e33200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:43:43.334 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:43.123+0000 7f51c8e33200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:43:43.334 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:43.333+0000 7f51c8e33200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:43:43.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:43.492+0000 7fd74e512200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-10T05:43:43.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:43.601+0000 7fd74e512200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: Standby manager daemon z restarted 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: Standby manager daemon z started 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: Standby manager daemon x restarted 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: Standby manager daemon x started 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:43.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:43:43.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: Standby manager daemon z restarted 2024-02-10T05:43:43.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: Standby manager daemon z started 2024-02-10T05:43:43.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-10T05:43:43.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: Standby manager daemon x restarted 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: Standby manager daemon x started 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: [10/Feb/2024:05:43:43] ENGINE Bus STARTING 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: CherryPy Checker: 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: The Application mounted at '' has an empty config. 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: 2024-02-10T05:43:43.660 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: [10/Feb/2024:05:43:43] ENGINE Serving on http://:::9283 2024-02-10T05:43:43.661 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:43:43 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x[31354]: [10/Feb/2024:05:43:43] ENGINE Bus STARTED 2024-02-10T05:43:43.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: Standby manager daemon z restarted 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: Standby manager daemon z started 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.110:0/690577778' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: Standby manager daemon x restarted 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: Standby manager daemon x started 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-02-10T05:43:43.741 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:43.288+0000 7f7763781200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:43:43.742 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: [10/Feb/2024:05:43:43] ENGINE Bus STARTING 2024-02-10T05:43:43.742 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: CherryPy Checker: 2024-02-10T05:43:43.742 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: The Application mounted at '' has an empty config. 2024-02-10T05:43:43.742 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: 2024-02-10T05:43:43.742 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: [10/Feb/2024:05:43:43] ENGINE Serving on http://:::9283 2024-02-10T05:43:43.742 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:43:43 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z[33141]: [10/Feb/2024:05:43:43] ENGINE Bus STARTED 2024-02-10T05:43:43.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:43.825+0000 7fd74e512200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-10T05:43:44.365 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:43 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:43:43] ENGINE Bus STARTING 2024-02-10T05:43:44.365 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: CherryPy Checker: 2024-02-10T05:43:44.365 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: The Application mounted at '' has an empty config. 2024-02-10T05:43:44.365 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:43:44.365 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:43:44] ENGINE Serving on http://:::9283 2024-02-10T05:43:44.365 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:43:44] ENGINE Bus STARTED 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: Active manager daemon y restarted 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: Activating manager daemon y 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: osdmap e31: 3 total, 3 up, 3 in 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: mgrmap e19: y(active, starting, since 0.00958734s), standbys: x, z 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:44.631 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: Manager daemon y is now available 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:44.632 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:44.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:44.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:44.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:44 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:44.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:44.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: Active manager daemon y restarted 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: Activating manager daemon y 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: osdmap e31: 3 total, 3 up, 3 in 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: mgrmap e19: y(active, starting, since 0.00958734s), standbys: x, z 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:43:44.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: Manager daemon y is now available 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:44.661 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:44 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.? 172.21.15.105:0/2068624165' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: mgrmap e18: y(active, since 3m), standbys: x, z 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: Active manager daemon y restarted 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: Activating manager daemon y 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: osdmap e31: 3 total, 3 up, 3 in 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: mgrmap e19: y(active, starting, since 0.00958734s), standbys: x, z 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-10T05:43:44.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: Manager daemon y is now available 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:44.741 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-10T05:43:44.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:44.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:44.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-02-10T05:43:44.742 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:44 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-02-10T05:43:46.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: [10/Feb/2024:05:43:44] ENGINE Bus STARTING 2024-02-10T05:43:46.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: [10/Feb/2024:05:43:44] ENGINE Serving on http://172.21.15.29:8765 2024-02-10T05:43:46.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: mgrmap e20: y(active, since 1.0146s), standbys: x, z 2024-02-10T05:43:46.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:46.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: [10/Feb/2024:05:43:44] ENGINE Serving on https://172.21.15.29:7150 2024-02-10T05:43:46.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: [10/Feb/2024:05:43:44] ENGINE Bus STARTED 2024-02-10T05:43:46.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:46.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:45 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:46.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: [10/Feb/2024:05:43:44] ENGINE Bus STARTING 2024-02-10T05:43:46.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: [10/Feb/2024:05:43:44] ENGINE Serving on http://172.21.15.29:8765 2024-02-10T05:43:46.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: mgrmap e20: y(active, since 1.0146s), standbys: x, z 2024-02-10T05:43:46.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:46.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: [10/Feb/2024:05:43:44] ENGINE Serving on https://172.21.15.29:7150 2024-02-10T05:43:46.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: [10/Feb/2024:05:43:44] ENGINE Bus STARTED 2024-02-10T05:43:46.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:46.160 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:45 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:46.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: [10/Feb/2024:05:43:44] ENGINE Bus STARTING 2024-02-10T05:43:46.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: [10/Feb/2024:05:43:44] ENGINE Serving on http://172.21.15.29:8765 2024-02-10T05:43:46.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: mgrmap e20: y(active, since 1.0146s), standbys: x, z 2024-02-10T05:43:46.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: pgmap v3: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:46.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: [10/Feb/2024:05:43:44] ENGINE Serving on https://172.21.15.29:7150 2024-02-10T05:43:46.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: [10/Feb/2024:05:43:44] ENGINE Bus STARTED 2024-02-10T05:43:46.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:46.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:45 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:46.658 INFO:teuthology.orchestra.run.smithi029.stdout:34359738389 2024-02-10T05:43:46.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-10T05:43:47.282 INFO:teuthology.orchestra.run.smithi029.stdout:77309411339 2024-02-10T05:43:47.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-10T05:43:47.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:47 smithi029 ceph-mon[25692]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:47.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:47 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:47 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:47 smithi029 ceph-mon[25692]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-02-10T05:43:47.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:47 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:47 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:47 smithi105 ceph-mon[29854]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:47.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:47 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:47 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:47 smithi105 ceph-mon[29854]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-02-10T05:43:47.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:47 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:47 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:47 smithi110 ceph-mon[30869]: pgmap v4: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:43:47.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:47 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:47 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:47 smithi110 ceph-mon[30869]: mgrmap e21: y(active, since 2s), standbys: x, z 2024-02-10T05:43:47.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:47 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:47.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:47 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.440 INFO:teuthology.orchestra.run.smithi029.stdout:55834574864 2024-02-10T05:43:48.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-10T05:43:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:48.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:48 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:48 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:48 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:48.643 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:48 smithi029 ceph-mon[25692]: Adjusting osd_memory_target on smithi105 to 16064M 2024-02-10T05:43:48.643 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:48 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:48 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:48 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:48 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:48.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:48 smithi105 ceph-mon[29854]: Adjusting osd_memory_target on smithi105 to 16064M 2024-02-10T05:43:48.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:48 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:48 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:48 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:48.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:48 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:48.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:48 smithi110 ceph-mon[30869]: Adjusting osd_memory_target on smithi105 to 16064M 2024-02-10T05:43:48.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:48 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:49.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:49 smithi029 ceph-mon[25692]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-10T05:43:49.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:49 smithi029 ceph-mon[25692]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-02-10T05:43:49.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:49 smithi105 ceph-mon[29854]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-10T05:43:49.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:49 smithi105 ceph-mon[29854]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-02-10T05:43:49.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:49 smithi110 ceph-mon[30869]: pgmap v5: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 69 KiB/s rd, 0 B/s wr, 38 op/s 2024-02-10T05:43:49.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:49 smithi110 ceph-mon[30869]: mgrmap e22: y(active, since 4s), standbys: x, z 2024-02-10T05:43:51.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:50 smithi029 ceph-mon[25692]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-10T05:43:51.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:50 smithi105 ceph-mon[29854]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-10T05:43:51.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:50 smithi110 ceph-mon[30869]: pgmap v6: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-02-10T05:43:52.100 INFO:teuthology.orchestra.run.smithi029.stdout:34359738390 2024-02-10T05:43:52.384 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:43:52 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ::ffff:172.21.15.105 - - [10/Feb/2024:05:43:52] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:43:53.103 INFO:teuthology.orchestra.run.smithi029.stdout:77309411340 2024-02-10T05:43:53.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:52 smithi029 ceph-mon[25692]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-10T05:43:53.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:52 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/584796216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:43:53.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:52 smithi105 ceph-mon[29854]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-10T05:43:53.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:52 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/584796216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:43:53.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:52 smithi110 ceph-mon[30869]: pgmap v7: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-02-10T05:43:53.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:52 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/584796216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-10T05:43:53.293 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:53.330 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738389 got 34359738390 for osd.0 2024-02-10T05:43:53.330 DEBUG:teuthology.parallel:result is None 2024-02-10T05:43:54.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:53 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3250221379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:43:54.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:53 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:54.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:53 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:54.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:53 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3250221379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:43:54.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:53 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:54.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:53 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:54.193 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411339 got 77309411340 for osd.2 2024-02-10T05:43:54.193 DEBUG:teuthology.parallel:result is None 2024-02-10T05:43:54.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:53 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3250221379' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-10T05:43:54.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:53 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:54.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:53 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:55.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:54 smithi029 ceph-mon[25692]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-10T05:43:55.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:54 smithi105 ceph-mon[29854]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-10T05:43:55.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:54 smithi110 ceph-mon[30869]: pgmap v8: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-02-10T05:43:55.874 INFO:teuthology.orchestra.run.smithi029.stdout:55834574865 2024-02-10T05:43:56.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:56.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: Adjusting osd_memory_target on smithi029 to 16069M 2024-02-10T05:43:56.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: Updating smithi105:/etc/ceph/ceph.conf 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: Updating smithi110:/etc/ceph/ceph.conf 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.410 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:56 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/3428851566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:43:56.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: Adjusting osd_memory_target on smithi029 to 16069M 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: Updating smithi105:/etc/ceph/ceph.conf 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: Updating smithi110:/etc/ceph/ceph.conf 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:56 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/3428851566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:43:56.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.543 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: Adjusting osd_memory_target on smithi029 to 16069M 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: Updating smithi029:/etc/ceph/ceph.conf 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: Updating smithi105:/etc/ceph/ceph.conf 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: Updating smithi110:/etc/ceph/ceph.conf 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-10T05:43:56.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:56 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/3428851566' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-10T05:43:56.568 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574864 got 55834574865 for osd.1 2024-02-10T05:43:56.569 DEBUG:teuthology.parallel:result is None 2024-02-10T05:43:56.569 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-10T05:43:56.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:43:56.818 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:43:57.485 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.485 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.485 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.485 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:43:57.485 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.485 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.486 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:57 smithi105 ceph-mon[29854]: Deploying daemon alertmanager.a on smithi029 2024-02-10T05:43:57.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:57 smithi110 ceph-mon[30869]: Deploying daemon alertmanager.a on smithi029 2024-02-10T05:43:57.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: Updating smithi110:/etc/ceph/ceph.client.admin.keyring 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: pgmap v9: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: Updating smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: Updating smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: Updating smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:43:57.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:57 smithi029 ceph-mon[25692]: Deploying daemon alertmanager.a on smithi029 2024-02-10T05:43:59.183 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:43:59.186 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-02-10T05:43:59.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:59 smithi110 ceph-mon[30869]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T05:43:59.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:43:59 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:59.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:59 smithi029 ceph-mon[25692]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T05:43:59.635 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:43:59 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:43:59.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:59 smithi105 ceph-mon[29854]: pgmap v10: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-02-10T05:43:59.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:43:59 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:00.310 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":10,"stamp":"2024-02-10T05:43:57.846178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":104944,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":101819,"kb_avail":281118224,"statfs":{"total":287972524032,"available":287865061376,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":104262992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":96,"num_read_kb":206,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003069"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837882+0000","last_change":"2024-02-10T05:43:21.872241+0000","last_active":"2024-02-10T05:43:43.837882+0000","last_peered":"2024-02-10T05:43:43.837882+0000","last_clean":"2024-02-10T05:43:43.837882+0000","last_became_active":"2024-02-10T05:43:21.871100+0000","last_became_peered":"2024-02-10T05:43:21.871100+0000","last_unstale":"2024-02-10T05:43:43.837882+0000","last_undegraded":"2024-02-10T05:43:43.837882+0000","last_fullsized":"2024-02-10T05:43:43.837882+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:08:43.815719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834266+0000","last_change":"2024-02-10T05:43:19.863222+0000","last_active":"2024-02-10T05:43:43.834266+0000","last_peered":"2024-02-10T05:43:43.834266+0000","last_clean":"2024-02-10T05:43:43.834266+0000","last_became_active":"2024-02-10T05:43:19.862931+0000","last_became_peered":"2024-02-10T05:43:19.862931+0000","last_unstale":"2024-02-10T05:43:43.834266+0000","last_undegraded":"2024-02-10T05:43:43.834266+0000","last_fullsized":"2024-02-10T05:43:43.834266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:03:16.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835793+0000","last_change":"2024-02-10T05:43:15.833988+0000","last_active":"2024-02-10T05:43:43.835793+0000","last_peered":"2024-02-10T05:43:43.835793+0000","last_clean":"2024-02-10T05:43:43.835793+0000","last_became_active":"2024-02-10T05:43:15.833729+0000","last_became_peered":"2024-02-10T05:43:15.833729+0000","last_unstale":"2024-02-10T05:43:43.835793+0000","last_undegraded":"2024-02-10T05:43:43.835793+0000","last_fullsized":"2024-02-10T05:43:43.835793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:57:50.234625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836066+0000","last_change":"2024-02-10T05:43:17.853426+0000","last_active":"2024-02-10T05:43:43.836066+0000","last_peered":"2024-02-10T05:43:43.836066+0000","last_clean":"2024-02-10T05:43:43.836066+0000","last_became_active":"2024-02-10T05:43:17.852478+0000","last_became_peered":"2024-02-10T05:43:17.852478+0000","last_unstale":"2024-02-10T05:43:43.836066+0000","last_undegraded":"2024-02-10T05:43:43.836066+0000","last_fullsized":"2024-02-10T05:43:43.836066+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:51:16.793620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832742+0000","last_change":"2024-02-10T05:43:21.870013+0000","last_active":"2024-02-10T05:43:43.832742+0000","last_peered":"2024-02-10T05:43:43.832742+0000","last_clean":"2024-02-10T05:43:43.832742+0000","last_became_active":"2024-02-10T05:43:21.869439+0000","last_became_peered":"2024-02-10T05:43:21.869439+0000","last_unstale":"2024-02-10T05:43:43.832742+0000","last_undegraded":"2024-02-10T05:43:43.832742+0000","last_fullsized":"2024-02-10T05:43:43.832742+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:13.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837592+0000","last_change":"2024-02-10T05:43:19.865045+0000","last_active":"2024-02-10T05:43:43.837592+0000","last_peered":"2024-02-10T05:43:43.837592+0000","last_clean":"2024-02-10T05:43:43.837592+0000","last_became_active":"2024-02-10T05:43:19.864745+0000","last_became_peered":"2024-02-10T05:43:19.864745+0000","last_unstale":"2024-02-10T05:43:43.837592+0000","last_undegraded":"2024-02-10T05:43:43.837592+0000","last_fullsized":"2024-02-10T05:43:43.837592+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:15:22.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836200+0000","last_change":"2024-02-10T05:43:15.839201+0000","last_active":"2024-02-10T05:43:43.836200+0000","last_peered":"2024-02-10T05:43:43.836200+0000","last_clean":"2024-02-10T05:43:43.836200+0000","last_became_active":"2024-02-10T05:43:15.838892+0000","last_became_peered":"2024-02-10T05:43:15.838892+0000","last_unstale":"2024-02-10T05:43:43.836200+0000","last_undegraded":"2024-02-10T05:43:43.836200+0000","last_fullsized":"2024-02-10T05:43:43.836200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:56:56.952608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834413+0000","last_change":"2024-02-10T05:43:17.851962+0000","last_active":"2024-02-10T05:43:43.834413+0000","last_peered":"2024-02-10T05:43:43.834413+0000","last_clean":"2024-02-10T05:43:43.834413+0000","last_became_active":"2024-02-10T05:43:17.851311+0000","last_became_peered":"2024-02-10T05:43:17.851311+0000","last_unstale":"2024-02-10T05:43:43.834413+0000","last_undegraded":"2024-02-10T05:43:43.834413+0000","last_fullsized":"2024-02-10T05:43:43.834413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:22:33.071870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834453+0000","last_change":"2024-02-10T05:43:21.867242+0000","last_active":"2024-02-10T05:43:43.834453+0000","last_peered":"2024-02-10T05:43:43.834453+0000","last_clean":"2024-02-10T05:43:43.834453+0000","last_became_active":"2024-02-10T05:43:21.866985+0000","last_became_peered":"2024-02-10T05:43:21.866985+0000","last_unstale":"2024-02-10T05:43:43.834453+0000","last_undegraded":"2024-02-10T05:43:43.834453+0000","last_fullsized":"2024-02-10T05:43:43.834453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:59:28.325957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838356+0000","last_change":"2024-02-10T05:43:19.865367+0000","last_active":"2024-02-10T05:43:43.838356+0000","last_peered":"2024-02-10T05:43:43.838356+0000","last_clean":"2024-02-10T05:43:43.838356+0000","last_became_active":"2024-02-10T05:43:19.864900+0000","last_became_peered":"2024-02-10T05:43:19.864900+0000","last_unstale":"2024-02-10T05:43:43.838356+0000","last_undegraded":"2024-02-10T05:43:43.838356+0000","last_fullsized":"2024-02-10T05:43:43.838356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:31:12.980741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836771+0000","last_change":"2024-02-10T05:43:15.833456+0000","last_active":"2024-02-10T05:43:43.836771+0000","last_peered":"2024-02-10T05:43:43.836771+0000","last_clean":"2024-02-10T05:43:43.836771+0000","last_became_active":"2024-02-10T05:43:15.833147+0000","last_became_peered":"2024-02-10T05:43:15.833147+0000","last_unstale":"2024-02-10T05:43:43.836771+0000","last_undegraded":"2024-02-10T05:43:43.836771+0000","last_fullsized":"2024-02-10T05:43:43.836771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:08:23.944728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838957+0000","last_change":"2024-02-10T05:43:17.849331+0000","last_active":"2024-02-10T05:43:43.838957+0000","last_peered":"2024-02-10T05:43:43.838957+0000","last_clean":"2024-02-10T05:43:43.838957+0000","last_became_active":"2024-02-10T05:43:17.848820+0000","last_became_peered":"2024-02-10T05:43:17.848820+0000","last_unstale":"2024-02-10T05:43:43.838957+0000","last_undegraded":"2024-02-10T05:43:43.838957+0000","last_fullsized":"2024-02-10T05:43:43.838957+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:44:18.786618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837717+0000","last_change":"2024-02-10T05:43:21.871282+0000","last_active":"2024-02-10T05:43:43.837717+0000","last_peered":"2024-02-10T05:43:43.837717+0000","last_clean":"2024-02-10T05:43:43.837717+0000","last_became_active":"2024-02-10T05:43:21.870886+0000","last_became_peered":"2024-02-10T05:43:21.870886+0000","last_unstale":"2024-02-10T05:43:43.837717+0000","last_undegraded":"2024-02-10T05:43:43.837717+0000","last_fullsized":"2024-02-10T05:43:43.837717+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:24:36.858562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838140+0000","last_change":"2024-02-10T05:43:19.863993+0000","last_active":"2024-02-10T05:43:43.838140+0000","last_peered":"2024-02-10T05:43:43.838140+0000","last_clean":"2024-02-10T05:43:43.838140+0000","last_became_active":"2024-02-10T05:43:19.863568+0000","last_became_peered":"2024-02-10T05:43:19.863568+0000","last_unstale":"2024-02-10T05:43:43.838140+0000","last_undegraded":"2024-02-10T05:43:43.838140+0000","last_fullsized":"2024-02-10T05:43:43.838140+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:09:28.019230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836022+0000","last_change":"2024-02-10T05:43:15.834913+0000","last_active":"2024-02-10T05:43:43.836022+0000","last_peered":"2024-02-10T05:43:43.836022+0000","last_clean":"2024-02-10T05:43:43.836022+0000","last_became_active":"2024-02-10T05:43:15.834585+0000","last_became_peered":"2024-02-10T05:43:15.834585+0000","last_unstale":"2024-02-10T05:43:43.836022+0000","last_undegraded":"2024-02-10T05:43:43.836022+0000","last_fullsized":"2024-02-10T05:43:43.836022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:23:12.687985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836524+0000","last_change":"2024-02-10T05:43:17.848384+0000","last_active":"2024-02-10T05:43:43.836524+0000","last_peered":"2024-02-10T05:43:43.836524+0000","last_clean":"2024-02-10T05:43:43.836524+0000","last_became_active":"2024-02-10T05:43:17.848141+0000","last_became_peered":"2024-02-10T05:43:17.848141+0000","last_unstale":"2024-02-10T05:43:43.836524+0000","last_undegraded":"2024-02-10T05:43:43.836524+0000","last_fullsized":"2024-02-10T05:43:43.836524+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:24:56.008390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838770+0000","last_change":"2024-02-10T05:43:21.876006+0000","last_active":"2024-02-10T05:43:43.838770+0000","last_peered":"2024-02-10T05:43:43.838770+0000","last_clean":"2024-02-10T05:43:43.838770+0000","last_became_active":"2024-02-10T05:43:21.875260+0000","last_became_peered":"2024-02-10T05:43:21.875260+0000","last_unstale":"2024-02-10T05:43:43.838770+0000","last_undegraded":"2024-02-10T05:43:43.838770+0000","last_fullsized":"2024-02-10T05:43:43.838770+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:15:42.453719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835914+0000","last_change":"2024-02-10T05:43:19.863554+0000","last_active":"2024-02-10T05:43:43.835914+0000","last_peered":"2024-02-10T05:43:43.835914+0000","last_clean":"2024-02-10T05:43:43.835914+0000","last_became_active":"2024-02-10T05:43:19.863165+0000","last_became_peered":"2024-02-10T05:43:19.863165+0000","last_unstale":"2024-02-10T05:43:43.835914+0000","last_undegraded":"2024-02-10T05:43:43.835914+0000","last_fullsized":"2024-02-10T05:43:43.835914+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:45.416316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835679+0000","last_change":"2024-02-10T05:43:15.840096+0000","last_active":"2024-02-10T05:43:43.835679+0000","last_peered":"2024-02-10T05:43:43.835679+0000","last_clean":"2024-02-10T05:43:43.835679+0000","last_became_active":"2024-02-10T05:43:15.839803+0000","last_became_peered":"2024-02-10T05:43:15.839803+0000","last_unstale":"2024-02-10T05:43:43.835679+0000","last_undegraded":"2024-02-10T05:43:43.835679+0000","last_fullsized":"2024-02-10T05:43:43.835679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:29:05.116034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835290+0000","last_change":"2024-02-10T05:43:17.845515+0000","last_active":"2024-02-10T05:43:43.835290+0000","last_peered":"2024-02-10T05:43:43.835290+0000","last_clean":"2024-02-10T05:43:43.835290+0000","last_became_active":"2024-02-10T05:43:17.845252+0000","last_became_peered":"2024-02-10T05:43:17.845252+0000","last_unstale":"2024-02-10T05:43:43.835290+0000","last_undegraded":"2024-02-10T05:43:43.835290+0000","last_fullsized":"2024-02-10T05:43:43.835290+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:03:26.635415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833946+0000","last_change":"2024-02-10T05:43:21.873645+0000","last_active":"2024-02-10T05:43:43.833946+0000","last_peered":"2024-02-10T05:43:43.833946+0000","last_clean":"2024-02-10T05:43:43.833946+0000","last_became_active":"2024-02-10T05:43:21.873314+0000","last_became_peered":"2024-02-10T05:43:21.873314+0000","last_unstale":"2024-02-10T05:43:43.833946+0000","last_undegraded":"2024-02-10T05:43:43.833946+0000","last_fullsized":"2024-02-10T05:43:43.833946+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:44.626374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835933+0000","last_change":"2024-02-10T05:43:19.860666+0000","last_active":"2024-02-10T05:43:43.835933+0000","last_peered":"2024-02-10T05:43:43.835933+0000","last_clean":"2024-02-10T05:43:43.835933+0000","last_became_active":"2024-02-10T05:43:19.860352+0000","last_became_peered":"2024-02-10T05:43:19.860352+0000","last_unstale":"2024-02-10T05:43:43.835933+0000","last_undegraded":"2024-02-10T05:43:43.835933+0000","last_fullsized":"2024-02-10T05:43:43.835933+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:44:06.114742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837905+0000","last_change":"2024-02-10T05:43:15.836611+0000","last_active":"2024-02-10T05:43:43.837905+0000","last_peered":"2024-02-10T05:43:43.837905+0000","last_clean":"2024-02-10T05:43:43.837905+0000","last_became_active":"2024-02-10T05:43:15.835756+0000","last_became_peered":"2024-02-10T05:43:15.835756+0000","last_unstale":"2024-02-10T05:43:43.837905+0000","last_undegraded":"2024-02-10T05:43:43.837905+0000","last_fullsized":"2024-02-10T05:43:43.837905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:13:32.879508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835423+0000","last_change":"2024-02-10T05:43:17.845494+0000","last_active":"2024-02-10T05:43:43.835423+0000","last_peered":"2024-02-10T05:43:43.835423+0000","last_clean":"2024-02-10T05:43:43.835423+0000","last_became_active":"2024-02-10T05:43:17.845203+0000","last_became_peered":"2024-02-10T05:43:17.845203+0000","last_unstale":"2024-02-10T05:43:43.835423+0000","last_undegraded":"2024-02-10T05:43:43.835423+0000","last_fullsized":"2024-02-10T05:43:43.835423+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:11:47.644560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833289+0000","last_change":"2024-02-10T05:43:21.874211+0000","last_active":"2024-02-10T05:43:43.833289+0000","last_peered":"2024-02-10T05:43:43.833289+0000","last_clean":"2024-02-10T05:43:43.833289+0000","last_became_active":"2024-02-10T05:43:21.873697+0000","last_became_peered":"2024-02-10T05:43:21.873697+0000","last_unstale":"2024-02-10T05:43:43.833289+0000","last_undegraded":"2024-02-10T05:43:43.833289+0000","last_fullsized":"2024-02-10T05:43:43.833289+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:58:01.449250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880518+0000","last_change":"2024-02-10T05:43:19.863965+0000","last_active":"2024-02-10T05:43:45.880518+0000","last_peered":"2024-02-10T05:43:45.880518+0000","last_clean":"2024-02-10T05:43:45.880518+0000","last_became_active":"2024-02-10T05:43:19.863101+0000","last_became_peered":"2024-02-10T05:43:19.863101+0000","last_unstale":"2024-02-10T05:43:45.880518+0000","last_undegraded":"2024-02-10T05:43:45.880518+0000","last_fullsized":"2024-02-10T05:43:45.880518+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:07:50.872812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833196+0000","last_change":"2024-02-10T05:43:15.839814+0000","last_active":"2024-02-10T05:43:43.833196+0000","last_peered":"2024-02-10T05:43:43.833196+0000","last_clean":"2024-02-10T05:43:43.833196+0000","last_became_active":"2024-02-10T05:43:15.839494+0000","last_became_peered":"2024-02-10T05:43:15.839494+0000","last_unstale":"2024-02-10T05:43:43.833196+0000","last_undegraded":"2024-02-10T05:43:43.833196+0000","last_fullsized":"2024-02-10T05:43:43.833196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:49:25.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832543+0000","last_change":"2024-02-10T05:43:17.848217+0000","last_active":"2024-02-10T05:43:43.832543+0000","last_peered":"2024-02-10T05:43:43.832543+0000","last_clean":"2024-02-10T05:43:43.832543+0000","last_became_active":"2024-02-10T05:43:17.847845+0000","last_became_peered":"2024-02-10T05:43:17.847845+0000","last_unstale":"2024-02-10T05:43:43.832543+0000","last_undegraded":"2024-02-10T05:43:43.832543+0000","last_fullsized":"2024-02-10T05:43:43.832543+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:04:40.792743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839477+0000","last_change":"2024-02-10T05:43:21.875902+0000","last_active":"2024-02-10T05:43:43.839477+0000","last_peered":"2024-02-10T05:43:43.839477+0000","last_clean":"2024-02-10T05:43:43.839477+0000","last_became_active":"2024-02-10T05:43:21.875402+0000","last_became_peered":"2024-02-10T05:43:21.875402+0000","last_unstale":"2024-02-10T05:43:43.839477+0000","last_undegraded":"2024-02-10T05:43:43.839477+0000","last_fullsized":"2024-02-10T05:43:43.839477+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:27:50.960863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835709+0000","last_change":"2024-02-10T05:43:19.866672+0000","last_active":"2024-02-10T05:43:43.835709+0000","last_peered":"2024-02-10T05:43:43.835709+0000","last_clean":"2024-02-10T05:43:43.835709+0000","last_became_active":"2024-02-10T05:43:19.866325+0000","last_became_peered":"2024-02-10T05:43:19.866325+0000","last_unstale":"2024-02-10T05:43:43.835709+0000","last_undegraded":"2024-02-10T05:43:43.835709+0000","last_fullsized":"2024-02-10T05:43:43.835709+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:02:28.958489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834381+0000","last_change":"2024-02-10T05:43:15.835531+0000","last_active":"2024-02-10T05:43:43.834381+0000","last_peered":"2024-02-10T05:43:43.834381+0000","last_clean":"2024-02-10T05:43:43.834381+0000","last_became_active":"2024-02-10T05:43:15.835227+0000","last_became_peered":"2024-02-10T05:43:15.835227+0000","last_unstale":"2024-02-10T05:43:43.834381+0000","last_undegraded":"2024-02-10T05:43:43.834381+0000","last_fullsized":"2024-02-10T05:43:43.834381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:26:23.178542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832741+0000","last_change":"2024-02-10T05:43:17.847125+0000","last_active":"2024-02-10T05:43:43.832741+0000","last_peered":"2024-02-10T05:43:43.832741+0000","last_clean":"2024-02-10T05:43:43.832741+0000","last_became_active":"2024-02-10T05:43:17.846685+0000","last_became_peered":"2024-02-10T05:43:17.846685+0000","last_unstale":"2024-02-10T05:43:43.832741+0000","last_undegraded":"2024-02-10T05:43:43.832741+0000","last_fullsized":"2024-02-10T05:43:43.832741+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:24:39.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838064+0000","last_change":"2024-02-10T05:43:21.871436+0000","last_active":"2024-02-10T05:43:43.838064+0000","last_peered":"2024-02-10T05:43:43.838064+0000","last_clean":"2024-02-10T05:43:43.838064+0000","last_became_active":"2024-02-10T05:43:21.870762+0000","last_became_peered":"2024-02-10T05:43:21.870762+0000","last_unstale":"2024-02-10T05:43:43.838064+0000","last_undegraded":"2024-02-10T05:43:43.838064+0000","last_fullsized":"2024-02-10T05:43:43.838064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:40.297965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832630+0000","last_change":"2024-02-10T05:43:19.863252+0000","last_active":"2024-02-10T05:43:43.832630+0000","last_peered":"2024-02-10T05:43:43.832630+0000","last_clean":"2024-02-10T05:43:43.832630+0000","last_became_active":"2024-02-10T05:43:19.862577+0000","last_became_peered":"2024-02-10T05:43:19.862577+0000","last_unstale":"2024-02-10T05:43:43.832630+0000","last_undegraded":"2024-02-10T05:43:43.832630+0000","last_fullsized":"2024-02-10T05:43:43.832630+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:56:36.681677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840604+0000","last_change":"2024-02-10T05:43:15.835854+0000","last_active":"2024-02-10T05:43:43.840604+0000","last_peered":"2024-02-10T05:43:43.840604+0000","last_clean":"2024-02-10T05:43:43.840604+0000","last_became_active":"2024-02-10T05:43:15.835346+0000","last_became_peered":"2024-02-10T05:43:15.835346+0000","last_unstale":"2024-02-10T05:43:43.840604+0000","last_undegraded":"2024-02-10T05:43:43.840604+0000","last_fullsized":"2024-02-10T05:43:43.840604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:12:03.804758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840134+0000","last_change":"2024-02-10T05:43:17.849024+0000","last_active":"2024-02-10T05:43:43.840134+0000","last_peered":"2024-02-10T05:43:43.840134+0000","last_clean":"2024-02-10T05:43:43.840134+0000","last_became_active":"2024-02-10T05:43:17.848694+0000","last_became_peered":"2024-02-10T05:43:17.848694+0000","last_unstale":"2024-02-10T05:43:43.840134+0000","last_undegraded":"2024-02-10T05:43:43.840134+0000","last_fullsized":"2024-02-10T05:43:43.840134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:42:30.542808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837684+0000","last_change":"2024-02-10T05:43:21.870780+0000","last_active":"2024-02-10T05:43:43.837684+0000","last_peered":"2024-02-10T05:43:43.837684+0000","last_clean":"2024-02-10T05:43:43.837684+0000","last_became_active":"2024-02-10T05:43:21.870345+0000","last_became_peered":"2024-02-10T05:43:21.870345+0000","last_unstale":"2024-02-10T05:43:43.837684+0000","last_undegraded":"2024-02-10T05:43:43.837684+0000","last_fullsized":"2024-02-10T05:43:43.837684+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:52:06.205277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833698+0000","last_change":"2024-02-10T05:43:19.863926+0000","last_active":"2024-02-10T05:43:43.833698+0000","last_peered":"2024-02-10T05:43:43.833698+0000","last_clean":"2024-02-10T05:43:43.833698+0000","last_became_active":"2024-02-10T05:43:19.863395+0000","last_became_peered":"2024-02-10T05:43:19.863395+0000","last_unstale":"2024-02-10T05:43:43.833698+0000","last_undegraded":"2024-02-10T05:43:43.833698+0000","last_fullsized":"2024-02-10T05:43:43.833698+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:04.726027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839054+0000","last_change":"2024-02-10T05:43:15.837026+0000","last_active":"2024-02-10T05:43:43.839054+0000","last_peered":"2024-02-10T05:43:43.839054+0000","last_clean":"2024-02-10T05:43:43.839054+0000","last_became_active":"2024-02-10T05:43:15.835534+0000","last_became_peered":"2024-02-10T05:43:15.835534+0000","last_unstale":"2024-02-10T05:43:43.839054+0000","last_undegraded":"2024-02-10T05:43:43.839054+0000","last_fullsized":"2024-02-10T05:43:43.839054+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:23:45.140460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834590+0000","last_change":"2024-02-10T05:43:17.851665+0000","last_active":"2024-02-10T05:43:43.834590+0000","last_peered":"2024-02-10T05:43:43.834590+0000","last_clean":"2024-02-10T05:43:43.834590+0000","last_became_active":"2024-02-10T05:43:17.851159+0000","last_became_peered":"2024-02-10T05:43:17.851159+0000","last_unstale":"2024-02-10T05:43:43.834590+0000","last_undegraded":"2024-02-10T05:43:43.834590+0000","last_fullsized":"2024-02-10T05:43:43.834590+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:05:07.649648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834585+0000","last_change":"2024-02-10T05:43:21.867198+0000","last_active":"2024-02-10T05:43:43.834585+0000","last_peered":"2024-02-10T05:43:43.834585+0000","last_clean":"2024-02-10T05:43:43.834585+0000","last_became_active":"2024-02-10T05:43:21.866931+0000","last_became_peered":"2024-02-10T05:43:21.866931+0000","last_unstale":"2024-02-10T05:43:43.834585+0000","last_undegraded":"2024-02-10T05:43:43.834585+0000","last_fullsized":"2024-02-10T05:43:43.834585+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:16:52.506175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838512+0000","last_change":"2024-02-10T05:43:19.863776+0000","last_active":"2024-02-10T05:43:43.838512+0000","last_peered":"2024-02-10T05:43:43.838512+0000","last_clean":"2024-02-10T05:43:43.838512+0000","last_became_active":"2024-02-10T05:43:19.863387+0000","last_became_peered":"2024-02-10T05:43:19.863387+0000","last_unstale":"2024-02-10T05:43:43.838512+0000","last_undegraded":"2024-02-10T05:43:43.838512+0000","last_fullsized":"2024-02-10T05:43:43.838512+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:24:25.700935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836605+0000","last_change":"2024-02-10T05:43:15.839398+0000","last_active":"2024-02-10T05:43:43.836605+0000","last_peered":"2024-02-10T05:43:43.836605+0000","last_clean":"2024-02-10T05:43:43.836605+0000","last_became_active":"2024-02-10T05:43:15.839111+0000","last_became_peered":"2024-02-10T05:43:15.839111+0000","last_unstale":"2024-02-10T05:43:43.836605+0000","last_undegraded":"2024-02-10T05:43:43.836605+0000","last_fullsized":"2024-02-10T05:43:43.836605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:03:19.259772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839327+0000","last_change":"2024-02-10T05:43:17.849355+0000","last_active":"2024-02-10T05:43:43.839327+0000","last_peered":"2024-02-10T05:43:43.839327+0000","last_clean":"2024-02-10T05:43:43.839327+0000","last_became_active":"2024-02-10T05:43:17.848739+0000","last_became_peered":"2024-02-10T05:43:17.848739+0000","last_unstale":"2024-02-10T05:43:43.839327+0000","last_undegraded":"2024-02-10T05:43:43.839327+0000","last_fullsized":"2024-02-10T05:43:43.839327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:12:09.191879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834628+0000","last_change":"2024-02-10T05:43:21.867725+0000","last_active":"2024-02-10T05:43:43.834628+0000","last_peered":"2024-02-10T05:43:43.834628+0000","last_clean":"2024-02-10T05:43:43.834628+0000","last_became_active":"2024-02-10T05:43:21.867468+0000","last_became_peered":"2024-02-10T05:43:21.867468+0000","last_unstale":"2024-02-10T05:43:43.834628+0000","last_undegraded":"2024-02-10T05:43:43.834628+0000","last_fullsized":"2024-02-10T05:43:43.834628+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:46:02.598735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834348+0000","last_change":"2024-02-10T05:43:19.862873+0000","last_active":"2024-02-10T05:43:43.834348+0000","last_peered":"2024-02-10T05:43:43.834348+0000","last_clean":"2024-02-10T05:43:43.834348+0000","last_became_active":"2024-02-10T05:43:19.862321+0000","last_became_peered":"2024-02-10T05:43:19.862321+0000","last_unstale":"2024-02-10T05:43:43.834348+0000","last_undegraded":"2024-02-10T05:43:43.834348+0000","last_fullsized":"2024-02-10T05:43:43.834348+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:45:29.409258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839081+0000","last_change":"2024-02-10T05:43:15.836536+0000","last_active":"2024-02-10T05:43:43.839081+0000","last_peered":"2024-02-10T05:43:43.839081+0000","last_clean":"2024-02-10T05:43:43.839081+0000","last_became_active":"2024-02-10T05:43:15.835889+0000","last_became_peered":"2024-02-10T05:43:15.835889+0000","last_unstale":"2024-02-10T05:43:43.839081+0000","last_undegraded":"2024-02-10T05:43:43.839081+0000","last_fullsized":"2024-02-10T05:43:43.839081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:59:02.230786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836223+0000","last_change":"2024-02-10T05:43:17.848491+0000","last_active":"2024-02-10T05:43:43.836223+0000","last_peered":"2024-02-10T05:43:43.836223+0000","last_clean":"2024-02-10T05:43:43.836223+0000","last_became_active":"2024-02-10T05:43:17.848197+0000","last_became_peered":"2024-02-10T05:43:17.848197+0000","last_unstale":"2024-02-10T05:43:43.836223+0000","last_undegraded":"2024-02-10T05:43:43.836223+0000","last_fullsized":"2024-02-10T05:43:43.836223+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:23:40.148005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833809+0000","last_change":"2024-02-10T05:43:21.871893+0000","last_active":"2024-02-10T05:43:43.833809+0000","last_peered":"2024-02-10T05:43:43.833809+0000","last_clean":"2024-02-10T05:43:43.833809+0000","last_became_active":"2024-02-10T05:43:21.871512+0000","last_became_peered":"2024-02-10T05:43:21.871512+0000","last_unstale":"2024-02-10T05:43:43.833809+0000","last_undegraded":"2024-02-10T05:43:43.833809+0000","last_fullsized":"2024-02-10T05:43:43.833809+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:33:57.915840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880496+0000","last_change":"2024-02-10T05:43:19.863596+0000","last_active":"2024-02-10T05:43:45.880496+0000","last_peered":"2024-02-10T05:43:45.880496+0000","last_clean":"2024-02-10T05:43:45.880496+0000","last_became_active":"2024-02-10T05:43:19.863153+0000","last_became_peered":"2024-02-10T05:43:19.863153+0000","last_unstale":"2024-02-10T05:43:45.880496+0000","last_undegraded":"2024-02-10T05:43:45.880496+0000","last_fullsized":"2024-02-10T05:43:45.880496+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:16:55.682185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835561+0000","last_change":"2024-02-10T05:43:15.833511+0000","last_active":"2024-02-10T05:43:43.835561+0000","last_peered":"2024-02-10T05:43:43.835561+0000","last_clean":"2024-02-10T05:43:43.835561+0000","last_became_active":"2024-02-10T05:43:15.833193+0000","last_became_peered":"2024-02-10T05:43:15.833193+0000","last_unstale":"2024-02-10T05:43:43.835561+0000","last_undegraded":"2024-02-10T05:43:43.835561+0000","last_fullsized":"2024-02-10T05:43:43.835561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:33:21.514252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834624+0000","last_change":"2024-02-10T05:43:17.851625+0000","last_active":"2024-02-10T05:43:43.834624+0000","last_peered":"2024-02-10T05:43:43.834624+0000","last_clean":"2024-02-10T05:43:43.834624+0000","last_became_active":"2024-02-10T05:43:17.851182+0000","last_became_peered":"2024-02-10T05:43:17.851182+0000","last_unstale":"2024-02-10T05:43:43.834624+0000","last_undegraded":"2024-02-10T05:43:43.834624+0000","last_fullsized":"2024-02-10T05:43:43.834624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:49:19.257417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839247+0000","last_change":"2024-02-10T05:43:21.871005+0000","last_active":"2024-02-10T05:43:43.839247+0000","last_peered":"2024-02-10T05:43:43.839247+0000","last_clean":"2024-02-10T05:43:43.839247+0000","last_became_active":"2024-02-10T05:43:21.870156+0000","last_became_peered":"2024-02-10T05:43:21.870156+0000","last_unstale":"2024-02-10T05:43:43.839247+0000","last_undegraded":"2024-02-10T05:43:43.839247+0000","last_fullsized":"2024-02-10T05:43:43.839247+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:58:30.202762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880478+0000","last_change":"2024-02-10T05:43:19.860460+0000","last_active":"2024-02-10T05:43:45.880478+0000","last_peered":"2024-02-10T05:43:45.880478+0000","last_clean":"2024-02-10T05:43:45.880478+0000","last_became_active":"2024-02-10T05:43:19.859626+0000","last_became_peered":"2024-02-10T05:43:19.859626+0000","last_unstale":"2024-02-10T05:43:45.880478+0000","last_undegraded":"2024-02-10T05:43:45.880478+0000","last_fullsized":"2024-02-10T05:43:45.880478+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:40:16.427778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835012+0000","last_change":"2024-02-10T05:43:15.835372+0000","last_active":"2024-02-10T05:43:43.835012+0000","last_peered":"2024-02-10T05:43:43.835012+0000","last_clean":"2024-02-10T05:43:43.835012+0000","last_became_active":"2024-02-10T05:43:15.835085+0000","last_became_peered":"2024-02-10T05:43:15.835085+0000","last_unstale":"2024-02-10T05:43:43.835012+0000","last_undegraded":"2024-02-10T05:43:43.835012+0000","last_fullsized":"2024-02-10T05:43:43.835012+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:03:45.833811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838891+0000","last_change":"2024-02-10T05:43:17.848023+0000","last_active":"2024-02-10T05:43:43.838891+0000","last_peered":"2024-02-10T05:43:43.838891+0000","last_clean":"2024-02-10T05:43:43.838891+0000","last_became_active":"2024-02-10T05:43:17.847687+0000","last_became_peered":"2024-02-10T05:43:17.847687+0000","last_unstale":"2024-02-10T05:43:43.838891+0000","last_undegraded":"2024-02-10T05:43:43.838891+0000","last_fullsized":"2024-02-10T05:43:43.838891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:48:59.436064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838494+0000","last_change":"2024-02-10T05:43:21.875415+0000","last_active":"2024-02-10T05:43:43.838494+0000","last_peered":"2024-02-10T05:43:43.838494+0000","last_clean":"2024-02-10T05:43:43.838494+0000","last_became_active":"2024-02-10T05:43:21.875097+0000","last_became_peered":"2024-02-10T05:43:21.875097+0000","last_unstale":"2024-02-10T05:43:43.838494+0000","last_undegraded":"2024-02-10T05:43:43.838494+0000","last_fullsized":"2024-02-10T05:43:43.838494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:52:55.049234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834231+0000","last_change":"2024-02-10T05:43:19.863140+0000","last_active":"2024-02-10T05:43:43.834231+0000","last_peered":"2024-02-10T05:43:43.834231+0000","last_clean":"2024-02-10T05:43:43.834231+0000","last_became_active":"2024-02-10T05:43:19.862624+0000","last_became_peered":"2024-02-10T05:43:19.862624+0000","last_unstale":"2024-02-10T05:43:43.834231+0000","last_undegraded":"2024-02-10T05:43:43.834231+0000","last_fullsized":"2024-02-10T05:43:43.834231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:45:01.229918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837786+0000","last_change":"2024-02-10T05:43:15.837870+0000","last_active":"2024-02-10T05:43:43.837786+0000","last_peered":"2024-02-10T05:43:43.837786+0000","last_clean":"2024-02-10T05:43:43.837786+0000","last_became_active":"2024-02-10T05:43:15.836387+0000","last_became_peered":"2024-02-10T05:43:15.836387+0000","last_unstale":"2024-02-10T05:43:43.837786+0000","last_undegraded":"2024-02-10T05:43:43.837786+0000","last_fullsized":"2024-02-10T05:43:43.837786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:36:04.803582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837134+0000","last_change":"2024-02-10T05:43:17.849873+0000","last_active":"2024-02-10T05:43:43.837134+0000","last_peered":"2024-02-10T05:43:43.837134+0000","last_clean":"2024-02-10T05:43:43.837134+0000","last_became_active":"2024-02-10T05:43:17.849584+0000","last_became_peered":"2024-02-10T05:43:17.849584+0000","last_unstale":"2024-02-10T05:43:43.837134+0000","last_undegraded":"2024-02-10T05:43:43.837134+0000","last_fullsized":"2024-02-10T05:43:43.837134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:56:51.183859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833948+0000","last_change":"2024-02-10T05:43:21.873506+0000","last_active":"2024-02-10T05:43:43.833948+0000","last_peered":"2024-02-10T05:43:43.833948+0000","last_clean":"2024-02-10T05:43:43.833948+0000","last_became_active":"2024-02-10T05:43:21.873050+0000","last_became_peered":"2024-02-10T05:43:21.873050+0000","last_unstale":"2024-02-10T05:43:43.833948+0000","last_undegraded":"2024-02-10T05:43:43.833948+0000","last_fullsized":"2024-02-10T05:43:43.833948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:42:33.425765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835795+0000","last_change":"2024-02-10T05:43:19.867288+0000","last_active":"2024-02-10T05:43:43.835795+0000","last_peered":"2024-02-10T05:43:43.835795+0000","last_clean":"2024-02-10T05:43:43.835795+0000","last_became_active":"2024-02-10T05:43:19.866951+0000","last_became_peered":"2024-02-10T05:43:19.866951+0000","last_unstale":"2024-02-10T05:43:43.835795+0000","last_undegraded":"2024-02-10T05:43:43.835795+0000","last_fullsized":"2024-02-10T05:43:43.835795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:26:24.093630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834499+0000","last_change":"2024-02-10T05:43:15.835499+0000","last_active":"2024-02-10T05:43:43.834499+0000","last_peered":"2024-02-10T05:43:43.834499+0000","last_clean":"2024-02-10T05:43:43.834499+0000","last_became_active":"2024-02-10T05:43:15.835197+0000","last_became_peered":"2024-02-10T05:43:15.835197+0000","last_unstale":"2024-02-10T05:43:43.834499+0000","last_undegraded":"2024-02-10T05:43:43.834499+0000","last_fullsized":"2024-02-10T05:43:43.834499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:30.677909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834968+0000","last_change":"2024-02-10T05:43:17.848368+0000","last_active":"2024-02-10T05:43:43.834968+0000","last_peered":"2024-02-10T05:43:43.834968+0000","last_clean":"2024-02-10T05:43:43.834968+0000","last_became_active":"2024-02-10T05:43:17.847957+0000","last_became_peered":"2024-02-10T05:43:17.847957+0000","last_unstale":"2024-02-10T05:43:43.834968+0000","last_undegraded":"2024-02-10T05:43:43.834968+0000","last_fullsized":"2024-02-10T05:43:43.834968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:59:22.013861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835385+0000","last_change":"2024-02-10T05:43:21.870341+0000","last_active":"2024-02-10T05:43:43.835385+0000","last_peered":"2024-02-10T05:43:43.835385+0000","last_clean":"2024-02-10T05:43:43.835385+0000","last_became_active":"2024-02-10T05:43:21.870050+0000","last_became_peered":"2024-02-10T05:43:21.870050+0000","last_unstale":"2024-02-10T05:43:43.835385+0000","last_undegraded":"2024-02-10T05:43:43.835385+0000","last_fullsized":"2024-02-10T05:43:43.835385+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:01:23.584030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836966+0000","last_change":"2024-02-10T05:43:15.833202+0000","last_active":"2024-02-10T05:43:43.836966+0000","last_peered":"2024-02-10T05:43:43.836966+0000","last_clean":"2024-02-10T05:43:43.836966+0000","last_became_active":"2024-02-10T05:43:15.832908+0000","last_became_peered":"2024-02-10T05:43:15.832908+0000","last_unstale":"2024-02-10T05:43:43.836966+0000","last_undegraded":"2024-02-10T05:43:43.836966+0000","last_fullsized":"2024-02-10T05:43:43.836966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:59:22.599607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837235+0000","last_change":"2024-02-10T05:43:17.848205+0000","last_active":"2024-02-10T05:43:43.837235+0000","last_peered":"2024-02-10T05:43:43.837235+0000","last_clean":"2024-02-10T05:43:43.837235+0000","last_became_active":"2024-02-10T05:43:17.847892+0000","last_became_peered":"2024-02-10T05:43:17.847892+0000","last_unstale":"2024-02-10T05:43:43.837235+0000","last_undegraded":"2024-02-10T05:43:43.837235+0000","last_fullsized":"2024-02-10T05:43:43.837235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:16:22.430428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880572+0000","last_change":"2024-02-10T05:43:19.860719+0000","last_active":"2024-02-10T05:43:45.880572+0000","last_peered":"2024-02-10T05:43:45.880572+0000","last_clean":"2024-02-10T05:43:45.880572+0000","last_became_active":"2024-02-10T05:43:19.860476+0000","last_became_peered":"2024-02-10T05:43:19.860476+0000","last_unstale":"2024-02-10T05:43:45.880572+0000","last_undegraded":"2024-02-10T05:43:45.880572+0000","last_fullsized":"2024-02-10T05:43:45.880572+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:02:08.541506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832988+0000","last_change":"2024-02-10T05:43:21.869808+0000","last_active":"2024-02-10T05:43:43.832988+0000","last_peered":"2024-02-10T05:43:43.832988+0000","last_clean":"2024-02-10T05:43:43.832988+0000","last_became_active":"2024-02-10T05:43:21.869472+0000","last_became_peered":"2024-02-10T05:43:21.869472+0000","last_unstale":"2024-02-10T05:43:43.832988+0000","last_undegraded":"2024-02-10T05:43:43.832988+0000","last_fullsized":"2024-02-10T05:43:43.832988+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:51:34.890026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835653+0000","last_change":"2024-02-10T05:43:15.839144+0000","last_active":"2024-02-10T05:43:43.835653+0000","last_peered":"2024-02-10T05:43:43.835653+0000","last_clean":"2024-02-10T05:43:43.835653+0000","last_became_active":"2024-02-10T05:43:15.838774+0000","last_became_peered":"2024-02-10T05:43:15.838774+0000","last_unstale":"2024-02-10T05:43:43.835653+0000","last_undegraded":"2024-02-10T05:43:43.835653+0000","last_fullsized":"2024-02-10T05:43:43.835653+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:30:21.148223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":53,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880443+0000","last_change":"2024-02-10T05:43:17.848010+0000","last_active":"2024-02-10T05:43:45.880443+0000","last_peered":"2024-02-10T05:43:45.880443+0000","last_clean":"2024-02-10T05:43:45.880443+0000","last_became_active":"2024-02-10T05:43:17.847533+0000","last_became_peered":"2024-02-10T05:43:17.847533+0000","last_unstale":"2024-02-10T05:43:45.880443+0000","last_undegraded":"2024-02-10T05:43:45.880443+0000","last_fullsized":"2024-02-10T05:43:45.880443+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:50:06.009222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834635+0000","last_change":"2024-02-10T05:43:19.860420+0000","last_active":"2024-02-10T05:43:43.834635+0000","last_peered":"2024-02-10T05:43:43.834635+0000","last_clean":"2024-02-10T05:43:43.834635+0000","last_became_active":"2024-02-10T05:43:19.859621+0000","last_became_peered":"2024-02-10T05:43:19.859621+0000","last_unstale":"2024-02-10T05:43:43.834635+0000","last_undegraded":"2024-02-10T05:43:43.834635+0000","last_fullsized":"2024-02-10T05:43:43.834635+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:13:50.841249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833158+0000","last_change":"2024-02-10T05:43:21.873625+0000","last_active":"2024-02-10T05:43:43.833158+0000","last_peered":"2024-02-10T05:43:43.833158+0000","last_clean":"2024-02-10T05:43:43.833158+0000","last_became_active":"2024-02-10T05:43:21.873199+0000","last_became_peered":"2024-02-10T05:43:21.873199+0000","last_unstale":"2024-02-10T05:43:43.833158+0000","last_undegraded":"2024-02-10T05:43:43.833158+0000","last_fullsized":"2024-02-10T05:43:43.833158+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:12:06.568634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838101+0000","last_change":"2024-02-10T05:43:15.836200+0000","last_active":"2024-02-10T05:43:43.838101+0000","last_peered":"2024-02-10T05:43:43.838101+0000","last_clean":"2024-02-10T05:43:43.838101+0000","last_became_active":"2024-02-10T05:43:15.835429+0000","last_became_peered":"2024-02-10T05:43:15.835429+0000","last_unstale":"2024-02-10T05:43:43.838101+0000","last_undegraded":"2024-02-10T05:43:43.838101+0000","last_fullsized":"2024-02-10T05:43:43.838101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:14:11.657449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839246+0000","last_change":"2024-02-10T05:43:17.847802+0000","last_active":"2024-02-10T05:43:43.839246+0000","last_peered":"2024-02-10T05:43:43.839246+0000","last_clean":"2024-02-10T05:43:43.839246+0000","last_became_active":"2024-02-10T05:43:17.847498+0000","last_became_peered":"2024-02-10T05:43:17.847498+0000","last_unstale":"2024-02-10T05:43:43.839246+0000","last_undegraded":"2024-02-10T05:43:43.839246+0000","last_fullsized":"2024-02-10T05:43:43.839246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:41:45.330257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833911+0000","last_change":"2024-02-10T05:43:19.863110+0000","last_active":"2024-02-10T05:43:43.833911+0000","last_peered":"2024-02-10T05:43:43.833911+0000","last_clean":"2024-02-10T05:43:43.833911+0000","last_became_active":"2024-02-10T05:43:19.862480+0000","last_became_peered":"2024-02-10T05:43:19.862480+0000","last_unstale":"2024-02-10T05:43:43.833911+0000","last_undegraded":"2024-02-10T05:43:43.833911+0000","last_fullsized":"2024-02-10T05:43:43.833911+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:25.669292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836302+0000","last_change":"2024-02-10T05:43:21.874425+0000","last_active":"2024-02-10T05:43:43.836302+0000","last_peered":"2024-02-10T05:43:43.836302+0000","last_clean":"2024-02-10T05:43:43.836302+0000","last_became_active":"2024-02-10T05:43:21.874178+0000","last_became_peered":"2024-02-10T05:43:21.874178+0000","last_unstale":"2024-02-10T05:43:43.836302+0000","last_undegraded":"2024-02-10T05:43:43.836302+0000","last_fullsized":"2024-02-10T05:43:43.836302+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:36:34.050976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834105+0000","last_change":"2024-02-10T05:43:17.847871+0000","last_active":"2024-02-10T05:43:43.834105+0000","last_peered":"2024-02-10T05:43:43.834105+0000","last_clean":"2024-02-10T05:43:43.834105+0000","last_became_active":"2024-02-10T05:43:17.847584+0000","last_became_peered":"2024-02-10T05:43:17.847584+0000","last_unstale":"2024-02-10T05:43:43.834105+0000","last_undegraded":"2024-02-10T05:43:43.834105+0000","last_fullsized":"2024-02-10T05:43:43.834105+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:29:36.934501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833034+0000","last_change":"2024-02-10T05:43:15.832741+0000","last_active":"2024-02-10T05:43:43.833034+0000","last_peered":"2024-02-10T05:43:43.833034+0000","last_clean":"2024-02-10T05:43:43.833034+0000","last_became_active":"2024-02-10T05:43:15.832480+0000","last_became_peered":"2024-02-10T05:43:15.832480+0000","last_unstale":"2024-02-10T05:43:43.833034+0000","last_undegraded":"2024-02-10T05:43:43.833034+0000","last_fullsized":"2024-02-10T05:43:43.833034+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:34:56.042494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834541+0000","last_change":"2024-02-10T05:43:19.866739+0000","last_active":"2024-02-10T05:43:43.834541+0000","last_peered":"2024-02-10T05:43:43.834541+0000","last_clean":"2024-02-10T05:43:43.834541+0000","last_became_active":"2024-02-10T05:43:19.866422+0000","last_became_peered":"2024-02-10T05:43:19.866422+0000","last_unstale":"2024-02-10T05:43:43.834541+0000","last_undegraded":"2024-02-10T05:43:43.834541+0000","last_fullsized":"2024-02-10T05:43:43.834541+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:02:47.481107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835472+0000","last_change":"2024-02-10T05:43:21.870176+0000","last_active":"2024-02-10T05:43:43.835472+0000","last_peered":"2024-02-10T05:43:43.835472+0000","last_clean":"2024-02-10T05:43:43.835472+0000","last_became_active":"2024-02-10T05:43:21.869872+0000","last_became_peered":"2024-02-10T05:43:21.869872+0000","last_unstale":"2024-02-10T05:43:43.835472+0000","last_undegraded":"2024-02-10T05:43:43.835472+0000","last_fullsized":"2024-02-10T05:43:43.835472+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:31:57.366906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834231+0000","last_change":"2024-02-10T05:43:17.847724+0000","last_active":"2024-02-10T05:43:43.834231+0000","last_peered":"2024-02-10T05:43:43.834231+0000","last_clean":"2024-02-10T05:43:43.834231+0000","last_became_active":"2024-02-10T05:43:17.847166+0000","last_became_peered":"2024-02-10T05:43:17.847166+0000","last_unstale":"2024-02-10T05:43:43.834231+0000","last_undegraded":"2024-02-10T05:43:43.834231+0000","last_fullsized":"2024-02-10T05:43:43.834231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:20.119597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838741+0000","last_change":"2024-02-10T05:43:15.836528+0000","last_active":"2024-02-10T05:43:43.838741+0000","last_peered":"2024-02-10T05:43:43.838741+0000","last_clean":"2024-02-10T05:43:43.838741+0000","last_became_active":"2024-02-10T05:43:15.835877+0000","last_became_peered":"2024-02-10T05:43:15.835877+0000","last_unstale":"2024-02-10T05:43:43.838741+0000","last_undegraded":"2024-02-10T05:43:43.838741+0000","last_fullsized":"2024-02-10T05:43:43.838741+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:37:59.156888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.930772+0000","last_change":"2024-02-10T05:43:03.910671+0000","last_active":"2024-02-10T05:43:45.930772+0000","last_peered":"2024-02-10T05:43:45.930772+0000","last_clean":"2024-02-10T05:43:45.930772+0000","last_became_active":"2024-02-10T05:43:03.910326+0000","last_became_peered":"2024-02-10T05:43:03.910326+0000","last_unstale":"2024-02-10T05:43:45.930772+0000","last_undegraded":"2024-02-10T05:43:45.930772+0000","last_fullsized":"2024-02-10T05:43:45.930772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:02.902254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:02.902254+0000","last_clean_scrub_stamp":"2024-02-10T05:43:02.902254+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:57:11.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839686+0000","last_change":"2024-02-10T05:43:19.864376+0000","last_active":"2024-02-10T05:43:43.839686+0000","last_peered":"2024-02-10T05:43:43.839686+0000","last_clean":"2024-02-10T05:43:43.839686+0000","last_became_active":"2024-02-10T05:43:19.863867+0000","last_became_peered":"2024-02-10T05:43:19.863867+0000","last_unstale":"2024-02-10T05:43:43.839686+0000","last_undegraded":"2024-02-10T05:43:43.839686+0000","last_fullsized":"2024-02-10T05:43:43.839686+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:05:04.553384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839688+0000","last_change":"2024-02-10T05:43:21.875731+0000","last_active":"2024-02-10T05:43:43.839688+0000","last_peered":"2024-02-10T05:43:43.839688+0000","last_clean":"2024-02-10T05:43:43.839688+0000","last_became_active":"2024-02-10T05:43:21.874946+0000","last_became_peered":"2024-02-10T05:43:21.874946+0000","last_unstale":"2024-02-10T05:43:43.839688+0000","last_undegraded":"2024-02-10T05:43:43.839688+0000","last_fullsized":"2024-02-10T05:43:43.839688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:34:52.991754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833314+0000","last_change":"2024-02-10T05:43:17.851641+0000","last_active":"2024-02-10T05:43:43.833314+0000","last_peered":"2024-02-10T05:43:43.833314+0000","last_clean":"2024-02-10T05:43:43.833314+0000","last_became_active":"2024-02-10T05:43:17.851257+0000","last_became_peered":"2024-02-10T05:43:17.851257+0000","last_unstale":"2024-02-10T05:43:43.833314+0000","last_undegraded":"2024-02-10T05:43:43.833314+0000","last_fullsized":"2024-02-10T05:43:43.833314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:44:17.022099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837162+0000","last_change":"2024-02-10T05:43:15.837997+0000","last_active":"2024-02-10T05:43:43.837162+0000","last_peered":"2024-02-10T05:43:43.837162+0000","last_clean":"2024-02-10T05:43:43.837162+0000","last_became_active":"2024-02-10T05:43:15.837364+0000","last_became_peered":"2024-02-10T05:43:15.837364+0000","last_unstale":"2024-02-10T05:43:43.837162+0000","last_undegraded":"2024-02-10T05:43:43.837162+0000","last_fullsized":"2024-02-10T05:43:43.837162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:38:22.913893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836690+0000","last_change":"2024-02-10T05:43:19.863341+0000","last_active":"2024-02-10T05:43:43.836690+0000","last_peered":"2024-02-10T05:43:43.836690+0000","last_clean":"2024-02-10T05:43:43.836690+0000","last_became_active":"2024-02-10T05:43:19.863055+0000","last_became_peered":"2024-02-10T05:43:19.863055+0000","last_unstale":"2024-02-10T05:43:43.836690+0000","last_undegraded":"2024-02-10T05:43:43.836690+0000","last_fullsized":"2024-02-10T05:43:43.836690+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:00:12.233736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837797+0000","last_change":"2024-02-10T05:43:21.875911+0000","last_active":"2024-02-10T05:43:43.837797+0000","last_peered":"2024-02-10T05:43:43.837797+0000","last_clean":"2024-02-10T05:43:43.837797+0000","last_became_active":"2024-02-10T05:43:21.875027+0000","last_became_peered":"2024-02-10T05:43:21.875027+0000","last_unstale":"2024-02-10T05:43:43.837797+0000","last_undegraded":"2024-02-10T05:43:43.837797+0000","last_fullsized":"2024-02-10T05:43:43.837797+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:27:02.546201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836391+0000","last_change":"2024-02-10T05:43:15.839090+0000","last_active":"2024-02-10T05:43:43.836391+0000","last_peered":"2024-02-10T05:43:43.836391+0000","last_clean":"2024-02-10T05:43:43.836391+0000","last_became_active":"2024-02-10T05:43:15.838668+0000","last_became_peered":"2024-02-10T05:43:15.838668+0000","last_unstale":"2024-02-10T05:43:43.836391+0000","last_undegraded":"2024-02-10T05:43:43.836391+0000","last_fullsized":"2024-02-10T05:43:43.836391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:14:42.909340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":91,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880439+0000","last_change":"2024-02-10T05:43:17.853328+0000","last_active":"2024-02-10T05:43:45.880439+0000","last_peered":"2024-02-10T05:43:45.880439+0000","last_clean":"2024-02-10T05:43:45.880439+0000","last_became_active":"2024-02-10T05:43:17.852388+0000","last_became_peered":"2024-02-10T05:43:17.852388+0000","last_unstale":"2024-02-10T05:43:45.880439+0000","last_undegraded":"2024-02-10T05:43:45.880439+0000","last_fullsized":"2024-02-10T05:43:45.880439+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:34:37.717866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880502+0000","last_change":"2024-02-10T05:43:19.860396+0000","last_active":"2024-02-10T05:43:45.880502+0000","last_peered":"2024-02-10T05:43:45.880502+0000","last_clean":"2024-02-10T05:43:45.880502+0000","last_became_active":"2024-02-10T05:43:19.859572+0000","last_became_peered":"2024-02-10T05:43:19.859572+0000","last_unstale":"2024-02-10T05:43:45.880502+0000","last_undegraded":"2024-02-10T05:43:45.880502+0000","last_fullsized":"2024-02-10T05:43:45.880502+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:06:57.757481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837199+0000","last_change":"2024-02-10T05:43:21.875794+0000","last_active":"2024-02-10T05:43:43.837199+0000","last_peered":"2024-02-10T05:43:43.837199+0000","last_clean":"2024-02-10T05:43:43.837199+0000","last_became_active":"2024-02-10T05:43:21.875065+0000","last_became_peered":"2024-02-10T05:43:21.875065+0000","last_unstale":"2024-02-10T05:43:43.837199+0000","last_undegraded":"2024-02-10T05:43:43.837199+0000","last_fullsized":"2024-02-10T05:43:43.837199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:57:18.944049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836302+0000","last_change":"2024-02-10T05:43:15.832744+0000","last_active":"2024-02-10T05:43:43.836302+0000","last_peered":"2024-02-10T05:43:43.836302+0000","last_clean":"2024-02-10T05:43:43.836302+0000","last_became_active":"2024-02-10T05:43:15.832523+0000","last_became_peered":"2024-02-10T05:43:15.832523+0000","last_unstale":"2024-02-10T05:43:43.836302+0000","last_undegraded":"2024-02-10T05:43:43.836302+0000","last_fullsized":"2024-02-10T05:43:43.836302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:28:33.379519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'21","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835951+0000","last_change":"2024-02-10T05:43:17.851969+0000","last_active":"2024-02-10T05:43:43.835951+0000","last_peered":"2024-02-10T05:43:43.835951+0000","last_clean":"2024-02-10T05:43:43.835951+0000","last_became_active":"2024-02-10T05:43:17.851660+0000","last_became_peered":"2024-02-10T05:43:17.851660+0000","last_unstale":"2024-02-10T05:43:43.835951+0000","last_undegraded":"2024-02-10T05:43:43.835951+0000","last_fullsized":"2024-02-10T05:43:43.835951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:57:27.504325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880445+0000","last_change":"2024-02-10T05:43:19.863492+0000","last_active":"2024-02-10T05:43:45.880445+0000","last_peered":"2024-02-10T05:43:45.880445+0000","last_clean":"2024-02-10T05:43:45.880445+0000","last_became_active":"2024-02-10T05:43:19.863149+0000","last_became_peered":"2024-02-10T05:43:19.863149+0000","last_unstale":"2024-02-10T05:43:45.880445+0000","last_undegraded":"2024-02-10T05:43:45.880445+0000","last_fullsized":"2024-02-10T05:43:45.880445+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:59:28.219067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835349+0000","last_change":"2024-02-10T05:43:21.870225+0000","last_active":"2024-02-10T05:43:43.835349+0000","last_peered":"2024-02-10T05:43:43.835349+0000","last_clean":"2024-02-10T05:43:43.835349+0000","last_became_active":"2024-02-10T05:43:21.869930+0000","last_became_peered":"2024-02-10T05:43:21.869930+0000","last_unstale":"2024-02-10T05:43:43.835349+0000","last_undegraded":"2024-02-10T05:43:43.835349+0000","last_fullsized":"2024-02-10T05:43:43.835349+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:15:20.444051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840351+0000","last_change":"2024-02-10T05:43:15.836108+0000","last_active":"2024-02-10T05:43:43.840351+0000","last_peered":"2024-02-10T05:43:43.840351+0000","last_clean":"2024-02-10T05:43:43.840351+0000","last_became_active":"2024-02-10T05:43:15.835598+0000","last_became_peered":"2024-02-10T05:43:15.835598+0000","last_unstale":"2024-02-10T05:43:43.840351+0000","last_undegraded":"2024-02-10T05:43:43.840351+0000","last_fullsized":"2024-02-10T05:43:43.840351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:02:07.036035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835941+0000","last_change":"2024-02-10T05:43:17.852433+0000","last_active":"2024-02-10T05:43:43.835941+0000","last_peered":"2024-02-10T05:43:43.835941+0000","last_clean":"2024-02-10T05:43:43.835941+0000","last_became_active":"2024-02-10T05:43:17.852124+0000","last_became_peered":"2024-02-10T05:43:17.852124+0000","last_unstale":"2024-02-10T05:43:43.835941+0000","last_undegraded":"2024-02-10T05:43:43.835941+0000","last_fullsized":"2024-02-10T05:43:43.835941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:13:26.424699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834511+0000","last_change":"2024-02-10T05:43:19.863338+0000","last_active":"2024-02-10T05:43:43.834511+0000","last_peered":"2024-02-10T05:43:43.834511+0000","last_clean":"2024-02-10T05:43:43.834511+0000","last_became_active":"2024-02-10T05:43:19.863028+0000","last_became_peered":"2024-02-10T05:43:19.863028+0000","last_unstale":"2024-02-10T05:43:43.834511+0000","last_undegraded":"2024-02-10T05:43:43.834511+0000","last_fullsized":"2024-02-10T05:43:43.834511+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:34:50.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839596+0000","last_change":"2024-02-10T05:43:21.875993+0000","last_active":"2024-02-10T05:43:43.839596+0000","last_peered":"2024-02-10T05:43:43.839596+0000","last_clean":"2024-02-10T05:43:43.839596+0000","last_became_active":"2024-02-10T05:43:21.875468+0000","last_became_peered":"2024-02-10T05:43:21.875468+0000","last_unstale":"2024-02-10T05:43:43.839596+0000","last_undegraded":"2024-02-10T05:43:43.839596+0000","last_fullsized":"2024-02-10T05:43:43.839596+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:47:46.160440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880408+0000","last_change":"2024-02-10T05:43:19.864323+0000","last_active":"2024-02-10T05:43:45.880408+0000","last_peered":"2024-02-10T05:43:45.880408+0000","last_clean":"2024-02-10T05:43:45.880408+0000","last_became_active":"2024-02-10T05:43:19.863883+0000","last_became_peered":"2024-02-10T05:43:19.863883+0000","last_unstale":"2024-02-10T05:43:45.880408+0000","last_undegraded":"2024-02-10T05:43:45.880408+0000","last_fullsized":"2024-02-10T05:43:45.880408+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:07:14.896925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837426+0000","last_change":"2024-02-10T05:43:15.837978+0000","last_active":"2024-02-10T05:43:43.837426+0000","last_peered":"2024-02-10T05:43:43.837426+0000","last_clean":"2024-02-10T05:43:43.837426+0000","last_became_active":"2024-02-10T05:43:15.836913+0000","last_became_peered":"2024-02-10T05:43:15.836913+0000","last_unstale":"2024-02-10T05:43:43.837426+0000","last_undegraded":"2024-02-10T05:43:43.837426+0000","last_fullsized":"2024-02-10T05:43:43.837426+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:21:11.634868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834812+0000","last_change":"2024-02-10T05:43:17.848295+0000","last_active":"2024-02-10T05:43:43.834812+0000","last_peered":"2024-02-10T05:43:43.834812+0000","last_clean":"2024-02-10T05:43:43.834812+0000","last_became_active":"2024-02-10T05:43:17.847924+0000","last_became_peered":"2024-02-10T05:43:17.847924+0000","last_unstale":"2024-02-10T05:43:43.834812+0000","last_undegraded":"2024-02-10T05:43:43.834812+0000","last_fullsized":"2024-02-10T05:43:43.834812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:06:22.286915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836781+0000","last_change":"2024-02-10T05:43:21.870347+0000","last_active":"2024-02-10T05:43:43.836781+0000","last_peered":"2024-02-10T05:43:43.836781+0000","last_clean":"2024-02-10T05:43:43.836781+0000","last_became_active":"2024-02-10T05:43:21.869996+0000","last_became_peered":"2024-02-10T05:43:21.869996+0000","last_unstale":"2024-02-10T05:43:43.836781+0000","last_undegraded":"2024-02-10T05:43:43.836781+0000","last_fullsized":"2024-02-10T05:43:43.836781+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:40:59.131735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834322+0000","last_change":"2024-02-10T05:43:19.864326+0000","last_active":"2024-02-10T05:43:43.834322+0000","last_peered":"2024-02-10T05:43:43.834322+0000","last_clean":"2024-02-10T05:43:43.834322+0000","last_became_active":"2024-02-10T05:43:19.863987+0000","last_became_peered":"2024-02-10T05:43:19.863987+0000","last_unstale":"2024-02-10T05:43:43.834322+0000","last_undegraded":"2024-02-10T05:43:43.834322+0000","last_fullsized":"2024-02-10T05:43:43.834322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:41:23.518744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832862+0000","last_change":"2024-02-10T05:43:15.840029+0000","last_active":"2024-02-10T05:43:43.832862+0000","last_peered":"2024-02-10T05:43:43.832862+0000","last_clean":"2024-02-10T05:43:43.832862+0000","last_became_active":"2024-02-10T05:43:15.839596+0000","last_became_peered":"2024-02-10T05:43:15.839596+0000","last_unstale":"2024-02-10T05:43:43.832862+0000","last_undegraded":"2024-02-10T05:43:43.832862+0000","last_fullsized":"2024-02-10T05:43:43.832862+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:28.254239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834608+0000","last_change":"2024-02-10T05:43:17.847867+0000","last_active":"2024-02-10T05:43:43.834608+0000","last_peered":"2024-02-10T05:43:43.834608+0000","last_clean":"2024-02-10T05:43:43.834608+0000","last_became_active":"2024-02-10T05:43:17.847582+0000","last_became_peered":"2024-02-10T05:43:17.847582+0000","last_unstale":"2024-02-10T05:43:43.834608+0000","last_undegraded":"2024-02-10T05:43:43.834608+0000","last_fullsized":"2024-02-10T05:43:43.834608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:16:06.397863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835802+0000","last_change":"2024-02-10T05:43:21.868219+0000","last_active":"2024-02-10T05:43:43.835802+0000","last_peered":"2024-02-10T05:43:43.835802+0000","last_clean":"2024-02-10T05:43:43.835802+0000","last_became_active":"2024-02-10T05:43:21.867956+0000","last_became_peered":"2024-02-10T05:43:21.867956+0000","last_unstale":"2024-02-10T05:43:43.835802+0000","last_undegraded":"2024-02-10T05:43:43.835802+0000","last_fullsized":"2024-02-10T05:43:43.835802+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:06:01.917026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835190+0000","last_change":"2024-02-10T05:43:15.833330+0000","last_active":"2024-02-10T05:43:43.835190+0000","last_peered":"2024-02-10T05:43:43.835190+0000","last_clean":"2024-02-10T05:43:43.835190+0000","last_became_active":"2024-02-10T05:43:15.833038+0000","last_became_peered":"2024-02-10T05:43:15.833038+0000","last_unstale":"2024-02-10T05:43:43.835190+0000","last_undegraded":"2024-02-10T05:43:43.835190+0000","last_fullsized":"2024-02-10T05:43:43.835190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:10.612538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834150+0000","last_change":"2024-02-10T05:43:17.848286+0000","last_active":"2024-02-10T05:43:43.834150+0000","last_peered":"2024-02-10T05:43:43.834150+0000","last_clean":"2024-02-10T05:43:43.834150+0000","last_became_active":"2024-02-10T05:43:17.847826+0000","last_became_peered":"2024-02-10T05:43:17.847826+0000","last_unstale":"2024-02-10T05:43:43.834150+0000","last_undegraded":"2024-02-10T05:43:43.834150+0000","last_fullsized":"2024-02-10T05:43:43.834150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:39:27.311005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838835+0000","last_change":"2024-02-10T05:43:19.863466+0000","last_active":"2024-02-10T05:43:43.838835+0000","last_peered":"2024-02-10T05:43:43.838835+0000","last_clean":"2024-02-10T05:43:43.838835+0000","last_became_active":"2024-02-10T05:43:19.863094+0000","last_became_peered":"2024-02-10T05:43:19.863094+0000","last_unstale":"2024-02-10T05:43:43.838835+0000","last_undegraded":"2024-02-10T05:43:43.838835+0000","last_fullsized":"2024-02-10T05:43:43.838835+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:39:32.002402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835489+0000","last_change":"2024-02-10T05:43:21.870233+0000","last_active":"2024-02-10T05:43:43.835489+0000","last_peered":"2024-02-10T05:43:43.835489+0000","last_clean":"2024-02-10T05:43:43.835489+0000","last_became_active":"2024-02-10T05:43:21.869926+0000","last_became_peered":"2024-02-10T05:43:21.869926+0000","last_unstale":"2024-02-10T05:43:43.835489+0000","last_undegraded":"2024-02-10T05:43:43.835489+0000","last_fullsized":"2024-02-10T05:43:43.835489+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:22:39.569819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":35,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.880373+0000","last_change":"2024-02-10T05:43:19.863670+0000","last_active":"2024-02-10T05:43:45.880373+0000","last_peered":"2024-02-10T05:43:45.880373+0000","last_clean":"2024-02-10T05:43:45.880373+0000","last_became_active":"2024-02-10T05:43:19.863344+0000","last_became_peered":"2024-02-10T05:43:19.863344+0000","last_unstale":"2024-02-10T05:43:45.880373+0000","last_undegraded":"2024-02-10T05:43:45.880373+0000","last_fullsized":"2024-02-10T05:43:45.880373+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:16:44.206305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835559+0000","last_change":"2024-02-10T05:43:15.839259+0000","last_active":"2024-02-10T05:43:43.835559+0000","last_peered":"2024-02-10T05:43:43.835559+0000","last_clean":"2024-02-10T05:43:43.835559+0000","last_became_active":"2024-02-10T05:43:15.838942+0000","last_became_peered":"2024-02-10T05:43:15.838942+0000","last_unstale":"2024-02-10T05:43:43.835559+0000","last_undegraded":"2024-02-10T05:43:43.835559+0000","last_fullsized":"2024-02-10T05:43:43.835559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:10:58.449047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838298+0000","last_change":"2024-02-10T05:43:17.849835+0000","last_active":"2024-02-10T05:43:43.838298+0000","last_peered":"2024-02-10T05:43:43.838298+0000","last_clean":"2024-02-10T05:43:43.838298+0000","last_became_active":"2024-02-10T05:43:17.849540+0000","last_became_peered":"2024-02-10T05:43:17.849540+0000","last_unstale":"2024-02-10T05:43:43.838298+0000","last_undegraded":"2024-02-10T05:43:43.838298+0000","last_fullsized":"2024-02-10T05:43:43.838298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:05:46.073612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834732+0000","last_change":"2024-02-10T05:43:21.868264+0000","last_active":"2024-02-10T05:43:43.834732+0000","last_peered":"2024-02-10T05:43:43.834732+0000","last_clean":"2024-02-10T05:43:43.834732+0000","last_became_active":"2024-02-10T05:43:21.867985+0000","last_became_peered":"2024-02-10T05:43:21.867985+0000","last_unstale":"2024-02-10T05:43:43.834732+0000","last_undegraded":"2024-02-10T05:43:43.834732+0000","last_fullsized":"2024-02-10T05:43:43.834732+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:23.863227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836156+0000","last_change":"2024-02-10T05:43:15.834373+0000","last_active":"2024-02-10T05:43:43.836156+0000","last_peered":"2024-02-10T05:43:43.836156+0000","last_clean":"2024-02-10T05:43:43.836156+0000","last_became_active":"2024-02-10T05:43:15.834049+0000","last_became_peered":"2024-02-10T05:43:15.834049+0000","last_unstale":"2024-02-10T05:43:43.836156+0000","last_undegraded":"2024-02-10T05:43:43.836156+0000","last_fullsized":"2024-02-10T05:43:43.836156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:59:30.697677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840413+0000","last_change":"2024-02-10T05:43:17.848298+0000","last_active":"2024-02-10T05:43:43.840413+0000","last_peered":"2024-02-10T05:43:43.840413+0000","last_clean":"2024-02-10T05:43:43.840413+0000","last_became_active":"2024-02-10T05:43:17.847854+0000","last_became_peered":"2024-02-10T05:43:17.847854+0000","last_unstale":"2024-02-10T05:43:43.840413+0000","last_undegraded":"2024-02-10T05:43:43.840413+0000","last_fullsized":"2024-02-10T05:43:43.840413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:12:12.480845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837625+0000","last_change":"2024-02-10T05:43:19.863557+0000","last_active":"2024-02-10T05:43:43.837625+0000","last_peered":"2024-02-10T05:43:43.837625+0000","last_clean":"2024-02-10T05:43:43.837625+0000","last_became_active":"2024-02-10T05:43:19.863269+0000","last_became_peered":"2024-02-10T05:43:19.863269+0000","last_unstale":"2024-02-10T05:43:43.837625+0000","last_undegraded":"2024-02-10T05:43:43.837625+0000","last_fullsized":"2024-02-10T05:43:43.837625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:08:22.931566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832726+0000","last_change":"2024-02-10T05:43:21.869921+0000","last_active":"2024-02-10T05:43:43.832726+0000","last_peered":"2024-02-10T05:43:43.832726+0000","last_clean":"2024-02-10T05:43:43.832726+0000","last_became_active":"2024-02-10T05:43:21.869608+0000","last_became_peered":"2024-02-10T05:43:21.869608+0000","last_unstale":"2024-02-10T05:43:43.832726+0000","last_undegraded":"2024-02-10T05:43:43.832726+0000","last_fullsized":"2024-02-10T05:43:43.832726+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:28:23.857899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836247+0000","last_change":"2024-02-10T05:43:15.839471+0000","last_active":"2024-02-10T05:43:43.836247+0000","last_peered":"2024-02-10T05:43:43.836247+0000","last_clean":"2024-02-10T05:43:43.836247+0000","last_became_active":"2024-02-10T05:43:15.838801+0000","last_became_peered":"2024-02-10T05:43:15.838801+0000","last_unstale":"2024-02-10T05:43:43.836247+0000","last_undegraded":"2024-02-10T05:43:43.836247+0000","last_fullsized":"2024-02-10T05:43:43.836247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:54:08.853699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839103+0000","last_change":"2024-02-10T05:43:17.848303+0000","last_active":"2024-02-10T05:43:43.839103+0000","last_peered":"2024-02-10T05:43:43.839103+0000","last_clean":"2024-02-10T05:43:43.839103+0000","last_became_active":"2024-02-10T05:43:17.847968+0000","last_became_peered":"2024-02-10T05:43:17.847968+0000","last_unstale":"2024-02-10T05:43:43.839103+0000","last_undegraded":"2024-02-10T05:43:43.839103+0000","last_fullsized":"2024-02-10T05:43:43.839103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:14:30.146762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833093+0000","last_change":"2024-02-10T05:43:19.862894+0000","last_active":"2024-02-10T05:43:43.833093+0000","last_peered":"2024-02-10T05:43:43.833093+0000","last_clean":"2024-02-10T05:43:43.833093+0000","last_became_active":"2024-02-10T05:43:19.862344+0000","last_became_peered":"2024-02-10T05:43:19.862344+0000","last_unstale":"2024-02-10T05:43:43.833093+0000","last_undegraded":"2024-02-10T05:43:43.833093+0000","last_fullsized":"2024-02-10T05:43:43.833093+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:52:16.872156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837262+0000","last_change":"2024-02-10T05:43:21.871037+0000","last_active":"2024-02-10T05:43:43.837262+0000","last_peered":"2024-02-10T05:43:43.837262+0000","last_clean":"2024-02-10T05:43:43.837262+0000","last_became_active":"2024-02-10T05:43:21.870074+0000","last_became_peered":"2024-02-10T05:43:21.870074+0000","last_unstale":"2024-02-10T05:43:43.837262+0000","last_undegraded":"2024-02-10T05:43:43.837262+0000","last_fullsized":"2024-02-10T05:43:43.837262+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:49:37.727636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836084+0000","last_change":"2024-02-10T05:43:15.834089+0000","last_active":"2024-02-10T05:43:43.836084+0000","last_peered":"2024-02-10T05:43:43.836084+0000","last_clean":"2024-02-10T05:43:43.836084+0000","last_became_active":"2024-02-10T05:43:15.833752+0000","last_became_peered":"2024-02-10T05:43:15.833752+0000","last_unstale":"2024-02-10T05:43:43.836084+0000","last_undegraded":"2024-02-10T05:43:43.836084+0000","last_fullsized":"2024-02-10T05:43:43.836084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:43:14.753366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840484+0000","last_change":"2024-02-10T05:43:17.848508+0000","last_active":"2024-02-10T05:43:43.840484+0000","last_peered":"2024-02-10T05:43:43.840484+0000","last_clean":"2024-02-10T05:43:43.840484+0000","last_became_active":"2024-02-10T05:43:17.848160+0000","last_became_peered":"2024-02-10T05:43:17.848160+0000","last_unstale":"2024-02-10T05:43:43.840484+0000","last_undegraded":"2024-02-10T05:43:43.840484+0000","last_fullsized":"2024-02-10T05:43:43.840484+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:26.986686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835104+0000","last_change":"2024-02-10T05:43:19.860956+0000","last_active":"2024-02-10T05:43:43.835104+0000","last_peered":"2024-02-10T05:43:43.835104+0000","last_clean":"2024-02-10T05:43:43.835104+0000","last_became_active":"2024-02-10T05:43:19.860684+0000","last_became_peered":"2024-02-10T05:43:19.860684+0000","last_unstale":"2024-02-10T05:43:43.835104+0000","last_undegraded":"2024-02-10T05:43:43.835104+0000","last_fullsized":"2024-02-10T05:43:43.835104+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:22:54.390879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":588,"ondisk_log_size":588,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411341,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704688,"statfs":{"total":95990841344,"available":95953600512,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738391,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704688,"statfs":{"total":95990841344,"available":95953600512,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.57999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.54100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:44:00.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-10T05:44:00.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:00 smithi110 ceph-mon[30869]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:44:00.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:00 smithi029 ceph-mon[25692]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:44:00.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:00 smithi105 ceph-mon[29854]: from='client.14511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:44:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:01.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:01 smithi110 ceph-mon[30869]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:01.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:01 smithi029 ceph-mon[25692]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:01.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:01 smithi105 ceph-mon[29854]: pgmap v11: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:02.252 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:02 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ::ffff:172.21.15.105 - - [10/Feb/2024:05:44:02] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:44:03.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:03 smithi110 ceph-mon[30869]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:03.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:03 smithi029 ceph-mon[25692]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:03.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:03 smithi105 ceph-mon[29854]: pgmap v12: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:05.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:05 smithi029 ceph-mon[25692]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:05.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:05 smithi110 ceph-mon[30869]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:05.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:05 smithi105 ceph-mon[29854]: pgmap v13: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:06.064 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:44:06.066 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-02-10T05:44:06.969 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:06 smithi029 systemd[1]: Starting Ceph alertmanager.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:07.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:07 smithi110 ceph-mon[30869]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:07.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:07 smithi110 ceph-mon[30869]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:44:07.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:07 smithi029 ceph-mon[25692]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:07.634 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:07 smithi029 ceph-mon[25692]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:44:07.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:07 smithi105 ceph-mon[29854]: pgmap v14: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:07.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:07 smithi105 ceph-mon[29854]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-10T05:44:08.186 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-02-10T05:44:05.848284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595908,"num_objects":228,"num_object_clones":0,"num_object_copies":684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":228,"num_whiteouts":0,"num_read":1262,"num_read_kb":1077,"num_write":889,"num_write_kb":4546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":387,"acting":387,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":387,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":281223168,"kb_used":104944,"kb_used_data":3084,"kb_used_omap":4,"kb_used_meta":101819,"kb_avail":281118224,"statfs":{"total":287972524032,"available":287865061376,"internally_reserved":0,"allocated":3158016,"data_stored":2100345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":104262992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003074"},"pg_stats":[{"pgid":"5.18","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837882+0000","last_change":"2024-02-10T05:43:21.872241+0000","last_active":"2024-02-10T05:43:43.837882+0000","last_peered":"2024-02-10T05:43:43.837882+0000","last_clean":"2024-02-10T05:43:43.837882+0000","last_became_active":"2024-02-10T05:43:21.871100+0000","last_became_peered":"2024-02-10T05:43:21.871100+0000","last_unstale":"2024-02-10T05:43:43.837882+0000","last_undegraded":"2024-02-10T05:43:43.837882+0000","last_fullsized":"2024-02-10T05:43:43.837882+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:08:43.815719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834266+0000","last_change":"2024-02-10T05:43:19.863222+0000","last_active":"2024-02-10T05:43:43.834266+0000","last_peered":"2024-02-10T05:43:43.834266+0000","last_clean":"2024-02-10T05:43:43.834266+0000","last_became_active":"2024-02-10T05:43:19.862931+0000","last_became_peered":"2024-02-10T05:43:19.862931+0000","last_unstale":"2024-02-10T05:43:43.834266+0000","last_undegraded":"2024-02-10T05:43:43.834266+0000","last_fullsized":"2024-02-10T05:43:43.834266+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:03:16.587481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835793+0000","last_change":"2024-02-10T05:43:15.833988+0000","last_active":"2024-02-10T05:43:43.835793+0000","last_peered":"2024-02-10T05:43:43.835793+0000","last_clean":"2024-02-10T05:43:43.835793+0000","last_became_active":"2024-02-10T05:43:15.833729+0000","last_became_peered":"2024-02-10T05:43:15.833729+0000","last_unstale":"2024-02-10T05:43:43.835793+0000","last_undegraded":"2024-02-10T05:43:43.835793+0000","last_fullsized":"2024-02-10T05:43:43.835793+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:57:50.234625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"30'16","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836066+0000","last_change":"2024-02-10T05:43:17.853426+0000","last_active":"2024-02-10T05:43:43.836066+0000","last_peered":"2024-02-10T05:43:43.836066+0000","last_clean":"2024-02-10T05:43:43.836066+0000","last_became_active":"2024-02-10T05:43:17.852478+0000","last_became_peered":"2024-02-10T05:43:17.852478+0000","last_unstale":"2024-02-10T05:43:43.836066+0000","last_undegraded":"2024-02-10T05:43:43.836066+0000","last_fullsized":"2024-02-10T05:43:43.836066+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:51:16.793620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832742+0000","last_change":"2024-02-10T05:43:21.870013+0000","last_active":"2024-02-10T05:43:43.832742+0000","last_peered":"2024-02-10T05:43:43.832742+0000","last_clean":"2024-02-10T05:43:43.832742+0000","last_became_active":"2024-02-10T05:43:21.869439+0000","last_became_peered":"2024-02-10T05:43:21.869439+0000","last_unstale":"2024-02-10T05:43:43.832742+0000","last_undegraded":"2024-02-10T05:43:43.832742+0000","last_fullsized":"2024-02-10T05:43:43.832742+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:13.823804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837592+0000","last_change":"2024-02-10T05:43:19.865045+0000","last_active":"2024-02-10T05:43:43.837592+0000","last_peered":"2024-02-10T05:43:43.837592+0000","last_clean":"2024-02-10T05:43:43.837592+0000","last_became_active":"2024-02-10T05:43:19.864745+0000","last_became_peered":"2024-02-10T05:43:19.864745+0000","last_unstale":"2024-02-10T05:43:43.837592+0000","last_undegraded":"2024-02-10T05:43:43.837592+0000","last_fullsized":"2024-02-10T05:43:43.837592+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:15:22.549917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836200+0000","last_change":"2024-02-10T05:43:15.839201+0000","last_active":"2024-02-10T05:43:43.836200+0000","last_peered":"2024-02-10T05:43:43.836200+0000","last_clean":"2024-02-10T05:43:43.836200+0000","last_became_active":"2024-02-10T05:43:15.838892+0000","last_became_peered":"2024-02-10T05:43:15.838892+0000","last_unstale":"2024-02-10T05:43:43.836200+0000","last_undegraded":"2024-02-10T05:43:43.836200+0000","last_fullsized":"2024-02-10T05:43:43.836200+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:56:56.952608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834413+0000","last_change":"2024-02-10T05:43:17.851962+0000","last_active":"2024-02-10T05:43:43.834413+0000","last_peered":"2024-02-10T05:43:43.834413+0000","last_clean":"2024-02-10T05:43:43.834413+0000","last_became_active":"2024-02-10T05:43:17.851311+0000","last_became_peered":"2024-02-10T05:43:17.851311+0000","last_unstale":"2024-02-10T05:43:43.834413+0000","last_undegraded":"2024-02-10T05:43:43.834413+0000","last_fullsized":"2024-02-10T05:43:43.834413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:22:33.071870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834453+0000","last_change":"2024-02-10T05:43:21.867242+0000","last_active":"2024-02-10T05:43:43.834453+0000","last_peered":"2024-02-10T05:43:43.834453+0000","last_clean":"2024-02-10T05:43:43.834453+0000","last_became_active":"2024-02-10T05:43:21.866985+0000","last_became_peered":"2024-02-10T05:43:21.866985+0000","last_unstale":"2024-02-10T05:43:43.834453+0000","last_undegraded":"2024-02-10T05:43:43.834453+0000","last_fullsized":"2024-02-10T05:43:43.834453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:59:28.325957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838356+0000","last_change":"2024-02-10T05:43:19.865367+0000","last_active":"2024-02-10T05:43:43.838356+0000","last_peered":"2024-02-10T05:43:43.838356+0000","last_clean":"2024-02-10T05:43:43.838356+0000","last_became_active":"2024-02-10T05:43:19.864900+0000","last_became_peered":"2024-02-10T05:43:19.864900+0000","last_unstale":"2024-02-10T05:43:43.838356+0000","last_undegraded":"2024-02-10T05:43:43.838356+0000","last_fullsized":"2024-02-10T05:43:43.838356+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:31:12.980741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836771+0000","last_change":"2024-02-10T05:43:15.833456+0000","last_active":"2024-02-10T05:43:43.836771+0000","last_peered":"2024-02-10T05:43:43.836771+0000","last_clean":"2024-02-10T05:43:43.836771+0000","last_became_active":"2024-02-10T05:43:15.833147+0000","last_became_peered":"2024-02-10T05:43:15.833147+0000","last_unstale":"2024-02-10T05:43:43.836771+0000","last_undegraded":"2024-02-10T05:43:43.836771+0000","last_fullsized":"2024-02-10T05:43:43.836771+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:08:23.944728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838957+0000","last_change":"2024-02-10T05:43:17.849331+0000","last_active":"2024-02-10T05:43:43.838957+0000","last_peered":"2024-02-10T05:43:43.838957+0000","last_clean":"2024-02-10T05:43:43.838957+0000","last_became_active":"2024-02-10T05:43:17.848820+0000","last_became_peered":"2024-02-10T05:43:17.848820+0000","last_unstale":"2024-02-10T05:43:43.838957+0000","last_undegraded":"2024-02-10T05:43:43.838957+0000","last_fullsized":"2024-02-10T05:43:43.838957+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:44:18.786618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837717+0000","last_change":"2024-02-10T05:43:21.871282+0000","last_active":"2024-02-10T05:43:43.837717+0000","last_peered":"2024-02-10T05:43:43.837717+0000","last_clean":"2024-02-10T05:43:43.837717+0000","last_became_active":"2024-02-10T05:43:21.870886+0000","last_became_peered":"2024-02-10T05:43:21.870886+0000","last_unstale":"2024-02-10T05:43:43.837717+0000","last_undegraded":"2024-02-10T05:43:43.837717+0000","last_fullsized":"2024-02-10T05:43:43.837717+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:24:36.858562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838140+0000","last_change":"2024-02-10T05:43:19.863993+0000","last_active":"2024-02-10T05:43:43.838140+0000","last_peered":"2024-02-10T05:43:43.838140+0000","last_clean":"2024-02-10T05:43:43.838140+0000","last_became_active":"2024-02-10T05:43:19.863568+0000","last_became_peered":"2024-02-10T05:43:19.863568+0000","last_unstale":"2024-02-10T05:43:43.838140+0000","last_undegraded":"2024-02-10T05:43:43.838140+0000","last_fullsized":"2024-02-10T05:43:43.838140+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:09:28.019230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836022+0000","last_change":"2024-02-10T05:43:15.834913+0000","last_active":"2024-02-10T05:43:43.836022+0000","last_peered":"2024-02-10T05:43:43.836022+0000","last_clean":"2024-02-10T05:43:43.836022+0000","last_became_active":"2024-02-10T05:43:15.834585+0000","last_became_peered":"2024-02-10T05:43:15.834585+0000","last_unstale":"2024-02-10T05:43:43.836022+0000","last_undegraded":"2024-02-10T05:43:43.836022+0000","last_fullsized":"2024-02-10T05:43:43.836022+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:23:12.687985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"30'18","reported_seq":58,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836524+0000","last_change":"2024-02-10T05:43:17.848384+0000","last_active":"2024-02-10T05:43:43.836524+0000","last_peered":"2024-02-10T05:43:43.836524+0000","last_clean":"2024-02-10T05:43:43.836524+0000","last_became_active":"2024-02-10T05:43:17.848141+0000","last_became_peered":"2024-02-10T05:43:17.848141+0000","last_unstale":"2024-02-10T05:43:43.836524+0000","last_undegraded":"2024-02-10T05:43:43.836524+0000","last_fullsized":"2024-02-10T05:43:43.836524+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:24:56.008390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838770+0000","last_change":"2024-02-10T05:43:21.876006+0000","last_active":"2024-02-10T05:43:43.838770+0000","last_peered":"2024-02-10T05:43:43.838770+0000","last_clean":"2024-02-10T05:43:43.838770+0000","last_became_active":"2024-02-10T05:43:21.875260+0000","last_became_peered":"2024-02-10T05:43:21.875260+0000","last_unstale":"2024-02-10T05:43:43.838770+0000","last_undegraded":"2024-02-10T05:43:43.838770+0000","last_fullsized":"2024-02-10T05:43:43.838770+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:15:42.453719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835914+0000","last_change":"2024-02-10T05:43:19.863554+0000","last_active":"2024-02-10T05:43:43.835914+0000","last_peered":"2024-02-10T05:43:43.835914+0000","last_clean":"2024-02-10T05:43:43.835914+0000","last_became_active":"2024-02-10T05:43:19.863165+0000","last_became_peered":"2024-02-10T05:43:19.863165+0000","last_unstale":"2024-02-10T05:43:43.835914+0000","last_undegraded":"2024-02-10T05:43:43.835914+0000","last_fullsized":"2024-02-10T05:43:43.835914+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:45.416316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835679+0000","last_change":"2024-02-10T05:43:15.840096+0000","last_active":"2024-02-10T05:43:43.835679+0000","last_peered":"2024-02-10T05:43:43.835679+0000","last_clean":"2024-02-10T05:43:43.835679+0000","last_became_active":"2024-02-10T05:43:15.839803+0000","last_became_peered":"2024-02-10T05:43:15.839803+0000","last_unstale":"2024-02-10T05:43:43.835679+0000","last_undegraded":"2024-02-10T05:43:43.835679+0000","last_fullsized":"2024-02-10T05:43:43.835679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:29:05.116034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835290+0000","last_change":"2024-02-10T05:43:17.845515+0000","last_active":"2024-02-10T05:43:43.835290+0000","last_peered":"2024-02-10T05:43:43.835290+0000","last_clean":"2024-02-10T05:43:43.835290+0000","last_became_active":"2024-02-10T05:43:17.845252+0000","last_became_peered":"2024-02-10T05:43:17.845252+0000","last_unstale":"2024-02-10T05:43:43.835290+0000","last_undegraded":"2024-02-10T05:43:43.835290+0000","last_fullsized":"2024-02-10T05:43:43.835290+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:03:26.635415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833946+0000","last_change":"2024-02-10T05:43:21.873645+0000","last_active":"2024-02-10T05:43:43.833946+0000","last_peered":"2024-02-10T05:43:43.833946+0000","last_clean":"2024-02-10T05:43:43.833946+0000","last_became_active":"2024-02-10T05:43:21.873314+0000","last_became_peered":"2024-02-10T05:43:21.873314+0000","last_unstale":"2024-02-10T05:43:43.833946+0000","last_undegraded":"2024-02-10T05:43:43.833946+0000","last_fullsized":"2024-02-10T05:43:43.833946+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:39:44.626374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835933+0000","last_change":"2024-02-10T05:43:19.860666+0000","last_active":"2024-02-10T05:43:43.835933+0000","last_peered":"2024-02-10T05:43:43.835933+0000","last_clean":"2024-02-10T05:43:43.835933+0000","last_became_active":"2024-02-10T05:43:19.860352+0000","last_became_peered":"2024-02-10T05:43:19.860352+0000","last_unstale":"2024-02-10T05:43:43.835933+0000","last_undegraded":"2024-02-10T05:43:43.835933+0000","last_fullsized":"2024-02-10T05:43:43.835933+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:44:06.114742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837905+0000","last_change":"2024-02-10T05:43:15.836611+0000","last_active":"2024-02-10T05:43:43.837905+0000","last_peered":"2024-02-10T05:43:43.837905+0000","last_clean":"2024-02-10T05:43:43.837905+0000","last_became_active":"2024-02-10T05:43:15.835756+0000","last_became_peered":"2024-02-10T05:43:15.835756+0000","last_unstale":"2024-02-10T05:43:43.837905+0000","last_undegraded":"2024-02-10T05:43:43.837905+0000","last_fullsized":"2024-02-10T05:43:43.837905+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:13:32.879508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"30'11","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835423+0000","last_change":"2024-02-10T05:43:17.845494+0000","last_active":"2024-02-10T05:43:43.835423+0000","last_peered":"2024-02-10T05:43:43.835423+0000","last_clean":"2024-02-10T05:43:43.835423+0000","last_became_active":"2024-02-10T05:43:17.845203+0000","last_became_peered":"2024-02-10T05:43:17.845203+0000","last_unstale":"2024-02-10T05:43:43.835423+0000","last_undegraded":"2024-02-10T05:43:43.835423+0000","last_fullsized":"2024-02-10T05:43:43.835423+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:11:47.644560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833289+0000","last_change":"2024-02-10T05:43:21.874211+0000","last_active":"2024-02-10T05:43:43.833289+0000","last_peered":"2024-02-10T05:43:43.833289+0000","last_clean":"2024-02-10T05:43:43.833289+0000","last_became_active":"2024-02-10T05:43:21.873697+0000","last_became_peered":"2024-02-10T05:43:21.873697+0000","last_unstale":"2024-02-10T05:43:43.833289+0000","last_undegraded":"2024-02-10T05:43:43.833289+0000","last_fullsized":"2024-02-10T05:43:43.833289+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:58:01.449250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882354+0000","last_change":"2024-02-10T05:43:19.863965+0000","last_active":"2024-02-10T05:43:55.882354+0000","last_peered":"2024-02-10T05:43:55.882354+0000","last_clean":"2024-02-10T05:43:55.882354+0000","last_became_active":"2024-02-10T05:43:19.863101+0000","last_became_peered":"2024-02-10T05:43:19.863101+0000","last_unstale":"2024-02-10T05:43:55.882354+0000","last_undegraded":"2024-02-10T05:43:55.882354+0000","last_fullsized":"2024-02-10T05:43:55.882354+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:07:50.872812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.19","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833196+0000","last_change":"2024-02-10T05:43:15.839814+0000","last_active":"2024-02-10T05:43:43.833196+0000","last_peered":"2024-02-10T05:43:43.833196+0000","last_clean":"2024-02-10T05:43:43.833196+0000","last_became_active":"2024-02-10T05:43:15.839494+0000","last_became_peered":"2024-02-10T05:43:15.839494+0000","last_unstale":"2024-02-10T05:43:43.833196+0000","last_undegraded":"2024-02-10T05:43:43.833196+0000","last_fullsized":"2024-02-10T05:43:43.833196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:49:25.528229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832543+0000","last_change":"2024-02-10T05:43:17.848217+0000","last_active":"2024-02-10T05:43:43.832543+0000","last_peered":"2024-02-10T05:43:43.832543+0000","last_clean":"2024-02-10T05:43:43.832543+0000","last_became_active":"2024-02-10T05:43:17.847845+0000","last_became_peered":"2024-02-10T05:43:17.847845+0000","last_unstale":"2024-02-10T05:43:43.832543+0000","last_undegraded":"2024-02-10T05:43:43.832543+0000","last_fullsized":"2024-02-10T05:43:43.832543+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:04:40.792743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839477+0000","last_change":"2024-02-10T05:43:21.875902+0000","last_active":"2024-02-10T05:43:43.839477+0000","last_peered":"2024-02-10T05:43:43.839477+0000","last_clean":"2024-02-10T05:43:43.839477+0000","last_became_active":"2024-02-10T05:43:21.875402+0000","last_became_peered":"2024-02-10T05:43:21.875402+0000","last_unstale":"2024-02-10T05:43:43.839477+0000","last_undegraded":"2024-02-10T05:43:43.839477+0000","last_fullsized":"2024-02-10T05:43:43.839477+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:27:50.960863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835709+0000","last_change":"2024-02-10T05:43:19.866672+0000","last_active":"2024-02-10T05:43:43.835709+0000","last_peered":"2024-02-10T05:43:43.835709+0000","last_clean":"2024-02-10T05:43:43.835709+0000","last_became_active":"2024-02-10T05:43:19.866325+0000","last_became_peered":"2024-02-10T05:43:19.866325+0000","last_unstale":"2024-02-10T05:43:43.835709+0000","last_undegraded":"2024-02-10T05:43:43.835709+0000","last_fullsized":"2024-02-10T05:43:43.835709+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:02:28.958489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834381+0000","last_change":"2024-02-10T05:43:15.835531+0000","last_active":"2024-02-10T05:43:43.834381+0000","last_peered":"2024-02-10T05:43:43.834381+0000","last_clean":"2024-02-10T05:43:43.834381+0000","last_became_active":"2024-02-10T05:43:15.835227+0000","last_became_peered":"2024-02-10T05:43:15.835227+0000","last_unstale":"2024-02-10T05:43:43.834381+0000","last_undegraded":"2024-02-10T05:43:43.834381+0000","last_fullsized":"2024-02-10T05:43:43.834381+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:26:23.178542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.19","version":"30'39","reported_seq":96,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832741+0000","last_change":"2024-02-10T05:43:17.847125+0000","last_active":"2024-02-10T05:43:43.832741+0000","last_peered":"2024-02-10T05:43:43.832741+0000","last_clean":"2024-02-10T05:43:43.832741+0000","last_became_active":"2024-02-10T05:43:17.846685+0000","last_became_peered":"2024-02-10T05:43:17.846685+0000","last_unstale":"2024-02-10T05:43:43.832741+0000","last_undegraded":"2024-02-10T05:43:43.832741+0000","last_fullsized":"2024-02-10T05:43:43.832741+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:24:39.694131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838064+0000","last_change":"2024-02-10T05:43:21.871436+0000","last_active":"2024-02-10T05:43:43.838064+0000","last_peered":"2024-02-10T05:43:43.838064+0000","last_clean":"2024-02-10T05:43:43.838064+0000","last_became_active":"2024-02-10T05:43:21.870762+0000","last_became_peered":"2024-02-10T05:43:21.870762+0000","last_unstale":"2024-02-10T05:43:43.838064+0000","last_undegraded":"2024-02-10T05:43:43.838064+0000","last_fullsized":"2024-02-10T05:43:43.838064+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:40.297965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832630+0000","last_change":"2024-02-10T05:43:19.863252+0000","last_active":"2024-02-10T05:43:43.832630+0000","last_peered":"2024-02-10T05:43:43.832630+0000","last_clean":"2024-02-10T05:43:43.832630+0000","last_became_active":"2024-02-10T05:43:19.862577+0000","last_became_peered":"2024-02-10T05:43:19.862577+0000","last_unstale":"2024-02-10T05:43:43.832630+0000","last_undegraded":"2024-02-10T05:43:43.832630+0000","last_fullsized":"2024-02-10T05:43:43.832630+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:56:36.681677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840604+0000","last_change":"2024-02-10T05:43:15.835854+0000","last_active":"2024-02-10T05:43:43.840604+0000","last_peered":"2024-02-10T05:43:43.840604+0000","last_clean":"2024-02-10T05:43:43.840604+0000","last_became_active":"2024-02-10T05:43:15.835346+0000","last_became_peered":"2024-02-10T05:43:15.835346+0000","last_unstale":"2024-02-10T05:43:43.840604+0000","last_undegraded":"2024-02-10T05:43:43.840604+0000","last_fullsized":"2024-02-10T05:43:43.840604+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:12:03.804758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"30'21","reported_seq":63,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840134+0000","last_change":"2024-02-10T05:43:17.849024+0000","last_active":"2024-02-10T05:43:43.840134+0000","last_peered":"2024-02-10T05:43:43.840134+0000","last_clean":"2024-02-10T05:43:43.840134+0000","last_became_active":"2024-02-10T05:43:17.848694+0000","last_became_peered":"2024-02-10T05:43:17.848694+0000","last_unstale":"2024-02-10T05:43:43.840134+0000","last_undegraded":"2024-02-10T05:43:43.840134+0000","last_fullsized":"2024-02-10T05:43:43.840134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:42:30.542808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.11","version":"30'1","reported_seq":16,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837684+0000","last_change":"2024-02-10T05:43:21.870780+0000","last_active":"2024-02-10T05:43:43.837684+0000","last_peered":"2024-02-10T05:43:43.837684+0000","last_clean":"2024-02-10T05:43:43.837684+0000","last_became_active":"2024-02-10T05:43:21.870345+0000","last_became_peered":"2024-02-10T05:43:21.870345+0000","last_unstale":"2024-02-10T05:43:43.837684+0000","last_undegraded":"2024-02-10T05:43:43.837684+0000","last_fullsized":"2024-02-10T05:43:43.837684+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:52:06.205277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833698+0000","last_change":"2024-02-10T05:43:19.863926+0000","last_active":"2024-02-10T05:43:43.833698+0000","last_peered":"2024-02-10T05:43:43.833698+0000","last_clean":"2024-02-10T05:43:43.833698+0000","last_became_active":"2024-02-10T05:43:19.863395+0000","last_became_peered":"2024-02-10T05:43:19.863395+0000","last_unstale":"2024-02-10T05:43:43.833698+0000","last_undegraded":"2024-02-10T05:43:43.833698+0000","last_fullsized":"2024-02-10T05:43:43.833698+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:04.726027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839054+0000","last_change":"2024-02-10T05:43:15.837026+0000","last_active":"2024-02-10T05:43:43.839054+0000","last_peered":"2024-02-10T05:43:43.839054+0000","last_clean":"2024-02-10T05:43:43.839054+0000","last_became_active":"2024-02-10T05:43:15.835534+0000","last_became_peered":"2024-02-10T05:43:15.835534+0000","last_unstale":"2024-02-10T05:43:43.839054+0000","last_undegraded":"2024-02-10T05:43:43.839054+0000","last_fullsized":"2024-02-10T05:43:43.839054+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:23:45.140460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"30'12","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834590+0000","last_change":"2024-02-10T05:43:17.851665+0000","last_active":"2024-02-10T05:43:43.834590+0000","last_peered":"2024-02-10T05:43:43.834590+0000","last_clean":"2024-02-10T05:43:43.834590+0000","last_became_active":"2024-02-10T05:43:17.851159+0000","last_became_peered":"2024-02-10T05:43:17.851159+0000","last_unstale":"2024-02-10T05:43:43.834590+0000","last_undegraded":"2024-02-10T05:43:43.834590+0000","last_fullsized":"2024-02-10T05:43:43.834590+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:05:07.649648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834585+0000","last_change":"2024-02-10T05:43:21.867198+0000","last_active":"2024-02-10T05:43:43.834585+0000","last_peered":"2024-02-10T05:43:43.834585+0000","last_clean":"2024-02-10T05:43:43.834585+0000","last_became_active":"2024-02-10T05:43:21.866931+0000","last_became_peered":"2024-02-10T05:43:21.866931+0000","last_unstale":"2024-02-10T05:43:43.834585+0000","last_undegraded":"2024-02-10T05:43:43.834585+0000","last_fullsized":"2024-02-10T05:43:43.834585+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:16:52.506175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838512+0000","last_change":"2024-02-10T05:43:19.863776+0000","last_active":"2024-02-10T05:43:43.838512+0000","last_peered":"2024-02-10T05:43:43.838512+0000","last_clean":"2024-02-10T05:43:43.838512+0000","last_became_active":"2024-02-10T05:43:19.863387+0000","last_became_peered":"2024-02-10T05:43:19.863387+0000","last_unstale":"2024-02-10T05:43:43.838512+0000","last_undegraded":"2024-02-10T05:43:43.838512+0000","last_fullsized":"2024-02-10T05:43:43.838512+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:24:25.700935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836605+0000","last_change":"2024-02-10T05:43:15.839398+0000","last_active":"2024-02-10T05:43:43.836605+0000","last_peered":"2024-02-10T05:43:43.836605+0000","last_clean":"2024-02-10T05:43:43.836605+0000","last_became_active":"2024-02-10T05:43:15.839111+0000","last_became_peered":"2024-02-10T05:43:15.839111+0000","last_unstale":"2024-02-10T05:43:43.836605+0000","last_undegraded":"2024-02-10T05:43:43.836605+0000","last_fullsized":"2024-02-10T05:43:43.836605+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:03:19.259772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.14","version":"30'22","reported_seq":62,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839327+0000","last_change":"2024-02-10T05:43:17.849355+0000","last_active":"2024-02-10T05:43:43.839327+0000","last_peered":"2024-02-10T05:43:43.839327+0000","last_clean":"2024-02-10T05:43:43.839327+0000","last_became_active":"2024-02-10T05:43:17.848739+0000","last_became_peered":"2024-02-10T05:43:17.848739+0000","last_unstale":"2024-02-10T05:43:43.839327+0000","last_undegraded":"2024-02-10T05:43:43.839327+0000","last_fullsized":"2024-02-10T05:43:43.839327+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:12:09.191879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834628+0000","last_change":"2024-02-10T05:43:21.867725+0000","last_active":"2024-02-10T05:43:43.834628+0000","last_peered":"2024-02-10T05:43:43.834628+0000","last_clean":"2024-02-10T05:43:43.834628+0000","last_became_active":"2024-02-10T05:43:21.867468+0000","last_became_peered":"2024-02-10T05:43:21.867468+0000","last_unstale":"2024-02-10T05:43:43.834628+0000","last_undegraded":"2024-02-10T05:43:43.834628+0000","last_fullsized":"2024-02-10T05:43:43.834628+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:46:02.598735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834348+0000","last_change":"2024-02-10T05:43:19.862873+0000","last_active":"2024-02-10T05:43:43.834348+0000","last_peered":"2024-02-10T05:43:43.834348+0000","last_clean":"2024-02-10T05:43:43.834348+0000","last_became_active":"2024-02-10T05:43:19.862321+0000","last_became_peered":"2024-02-10T05:43:19.862321+0000","last_unstale":"2024-02-10T05:43:43.834348+0000","last_undegraded":"2024-02-10T05:43:43.834348+0000","last_fullsized":"2024-02-10T05:43:43.834348+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:45:29.409258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"23'1","reported_seq":33,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839081+0000","last_change":"2024-02-10T05:43:15.836536+0000","last_active":"2024-02-10T05:43:43.839081+0000","last_peered":"2024-02-10T05:43:43.839081+0000","last_clean":"2024-02-10T05:43:43.839081+0000","last_became_active":"2024-02-10T05:43:15.835889+0000","last_became_peered":"2024-02-10T05:43:15.835889+0000","last_unstale":"2024-02-10T05:43:43.839081+0000","last_undegraded":"2024-02-10T05:43:43.839081+0000","last_fullsized":"2024-02-10T05:43:43.839081+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:59:02.230786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836223+0000","last_change":"2024-02-10T05:43:17.848491+0000","last_active":"2024-02-10T05:43:43.836223+0000","last_peered":"2024-02-10T05:43:43.836223+0000","last_clean":"2024-02-10T05:43:43.836223+0000","last_became_active":"2024-02-10T05:43:17.848197+0000","last_became_peered":"2024-02-10T05:43:17.848197+0000","last_unstale":"2024-02-10T05:43:43.836223+0000","last_undegraded":"2024-02-10T05:43:43.836223+0000","last_fullsized":"2024-02-10T05:43:43.836223+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:23:40.148005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833809+0000","last_change":"2024-02-10T05:43:21.871893+0000","last_active":"2024-02-10T05:43:43.833809+0000","last_peered":"2024-02-10T05:43:43.833809+0000","last_clean":"2024-02-10T05:43:43.833809+0000","last_became_active":"2024-02-10T05:43:21.871512+0000","last_became_peered":"2024-02-10T05:43:21.871512+0000","last_unstale":"2024-02-10T05:43:43.833809+0000","last_undegraded":"2024-02-10T05:43:43.833809+0000","last_fullsized":"2024-02-10T05:43:43.833809+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:33:57.915840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882391+0000","last_change":"2024-02-10T05:43:19.863596+0000","last_active":"2024-02-10T05:43:55.882391+0000","last_peered":"2024-02-10T05:43:55.882391+0000","last_clean":"2024-02-10T05:43:55.882391+0000","last_became_active":"2024-02-10T05:43:19.863153+0000","last_became_peered":"2024-02-10T05:43:19.863153+0000","last_unstale":"2024-02-10T05:43:55.882391+0000","last_undegraded":"2024-02-10T05:43:55.882391+0000","last_fullsized":"2024-02-10T05:43:55.882391+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:16:55.682185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835561+0000","last_change":"2024-02-10T05:43:15.833511+0000","last_active":"2024-02-10T05:43:43.835561+0000","last_peered":"2024-02-10T05:43:43.835561+0000","last_clean":"2024-02-10T05:43:43.835561+0000","last_became_active":"2024-02-10T05:43:15.833193+0000","last_became_peered":"2024-02-10T05:43:15.833193+0000","last_unstale":"2024-02-10T05:43:43.835561+0000","last_undegraded":"2024-02-10T05:43:43.835561+0000","last_fullsized":"2024-02-10T05:43:43.835561+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:33:21.514252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"30'9","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834624+0000","last_change":"2024-02-10T05:43:17.851625+0000","last_active":"2024-02-10T05:43:43.834624+0000","last_peered":"2024-02-10T05:43:43.834624+0000","last_clean":"2024-02-10T05:43:43.834624+0000","last_became_active":"2024-02-10T05:43:17.851182+0000","last_became_peered":"2024-02-10T05:43:17.851182+0000","last_unstale":"2024-02-10T05:43:43.834624+0000","last_undegraded":"2024-02-10T05:43:43.834624+0000","last_fullsized":"2024-02-10T05:43:43.834624+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:49:19.257417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839247+0000","last_change":"2024-02-10T05:43:21.871005+0000","last_active":"2024-02-10T05:43:43.839247+0000","last_peered":"2024-02-10T05:43:43.839247+0000","last_clean":"2024-02-10T05:43:43.839247+0000","last_became_active":"2024-02-10T05:43:21.870156+0000","last_became_peered":"2024-02-10T05:43:21.870156+0000","last_unstale":"2024-02-10T05:43:43.839247+0000","last_undegraded":"2024-02-10T05:43:43.839247+0000","last_fullsized":"2024-02-10T05:43:43.839247+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:58:30.202762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.14","version":"30'8","reported_seq":40,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882441+0000","last_change":"2024-02-10T05:43:19.860460+0000","last_active":"2024-02-10T05:43:55.882441+0000","last_peered":"2024-02-10T05:43:55.882441+0000","last_clean":"2024-02-10T05:43:55.882441+0000","last_became_active":"2024-02-10T05:43:19.859626+0000","last_became_peered":"2024-02-10T05:43:19.859626+0000","last_unstale":"2024-02-10T05:43:55.882441+0000","last_undegraded":"2024-02-10T05:43:55.882441+0000","last_fullsized":"2024-02-10T05:43:55.882441+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:40:16.427778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835012+0000","last_change":"2024-02-10T05:43:15.835372+0000","last_active":"2024-02-10T05:43:43.835012+0000","last_peered":"2024-02-10T05:43:43.835012+0000","last_clean":"2024-02-10T05:43:43.835012+0000","last_became_active":"2024-02-10T05:43:15.835085+0000","last_became_peered":"2024-02-10T05:43:15.835085+0000","last_unstale":"2024-02-10T05:43:43.835012+0000","last_undegraded":"2024-02-10T05:43:43.835012+0000","last_fullsized":"2024-02-10T05:43:43.835012+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:03:45.833811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.13","version":"30'11","reported_seq":42,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838891+0000","last_change":"2024-02-10T05:43:17.848023+0000","last_active":"2024-02-10T05:43:43.838891+0000","last_peered":"2024-02-10T05:43:43.838891+0000","last_clean":"2024-02-10T05:43:43.838891+0000","last_became_active":"2024-02-10T05:43:17.847687+0000","last_became_peered":"2024-02-10T05:43:17.847687+0000","last_unstale":"2024-02-10T05:43:43.838891+0000","last_undegraded":"2024-02-10T05:43:43.838891+0000","last_fullsized":"2024-02-10T05:43:43.838891+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:48:59.436064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838494+0000","last_change":"2024-02-10T05:43:21.875415+0000","last_active":"2024-02-10T05:43:43.838494+0000","last_peered":"2024-02-10T05:43:43.838494+0000","last_clean":"2024-02-10T05:43:43.838494+0000","last_became_active":"2024-02-10T05:43:21.875097+0000","last_became_peered":"2024-02-10T05:43:21.875097+0000","last_unstale":"2024-02-10T05:43:43.838494+0000","last_undegraded":"2024-02-10T05:43:43.838494+0000","last_fullsized":"2024-02-10T05:43:43.838494+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:52:55.049234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834231+0000","last_change":"2024-02-10T05:43:19.863140+0000","last_active":"2024-02-10T05:43:43.834231+0000","last_peered":"2024-02-10T05:43:43.834231+0000","last_clean":"2024-02-10T05:43:43.834231+0000","last_became_active":"2024-02-10T05:43:19.862624+0000","last_became_peered":"2024-02-10T05:43:19.862624+0000","last_unstale":"2024-02-10T05:43:43.834231+0000","last_undegraded":"2024-02-10T05:43:43.834231+0000","last_fullsized":"2024-02-10T05:43:43.834231+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:45:01.229918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837786+0000","last_change":"2024-02-10T05:43:15.837870+0000","last_active":"2024-02-10T05:43:43.837786+0000","last_peered":"2024-02-10T05:43:43.837786+0000","last_clean":"2024-02-10T05:43:43.837786+0000","last_became_active":"2024-02-10T05:43:15.836387+0000","last_became_peered":"2024-02-10T05:43:15.836387+0000","last_unstale":"2024-02-10T05:43:43.837786+0000","last_undegraded":"2024-02-10T05:43:43.837786+0000","last_fullsized":"2024-02-10T05:43:43.837786+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:36:04.803582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"30'4","reported_seq":29,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837134+0000","last_change":"2024-02-10T05:43:17.849873+0000","last_active":"2024-02-10T05:43:43.837134+0000","last_peered":"2024-02-10T05:43:43.837134+0000","last_clean":"2024-02-10T05:43:43.837134+0000","last_became_active":"2024-02-10T05:43:17.849584+0000","last_became_peered":"2024-02-10T05:43:17.849584+0000","last_unstale":"2024-02-10T05:43:43.837134+0000","last_undegraded":"2024-02-10T05:43:43.837134+0000","last_fullsized":"2024-02-10T05:43:43.837134+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:56:51.183859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833948+0000","last_change":"2024-02-10T05:43:21.873506+0000","last_active":"2024-02-10T05:43:43.833948+0000","last_peered":"2024-02-10T05:43:43.833948+0000","last_clean":"2024-02-10T05:43:43.833948+0000","last_became_active":"2024-02-10T05:43:21.873050+0000","last_became_peered":"2024-02-10T05:43:21.873050+0000","last_unstale":"2024-02-10T05:43:43.833948+0000","last_undegraded":"2024-02-10T05:43:43.833948+0000","last_fullsized":"2024-02-10T05:43:43.833948+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:42:33.425765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835795+0000","last_change":"2024-02-10T05:43:19.867288+0000","last_active":"2024-02-10T05:43:43.835795+0000","last_peered":"2024-02-10T05:43:43.835795+0000","last_clean":"2024-02-10T05:43:43.835795+0000","last_became_active":"2024-02-10T05:43:19.866951+0000","last_became_peered":"2024-02-10T05:43:19.866951+0000","last_unstale":"2024-02-10T05:43:43.835795+0000","last_undegraded":"2024-02-10T05:43:43.835795+0000","last_fullsized":"2024-02-10T05:43:43.835795+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:26:24.093630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834499+0000","last_change":"2024-02-10T05:43:15.835499+0000","last_active":"2024-02-10T05:43:43.834499+0000","last_peered":"2024-02-10T05:43:43.834499+0000","last_clean":"2024-02-10T05:43:43.834499+0000","last_became_active":"2024-02-10T05:43:15.835197+0000","last_became_peered":"2024-02-10T05:43:15.835197+0000","last_unstale":"2024-02-10T05:43:43.834499+0000","last_undegraded":"2024-02-10T05:43:43.834499+0000","last_fullsized":"2024-02-10T05:43:43.834499+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:44:30.677909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834968+0000","last_change":"2024-02-10T05:43:17.848368+0000","last_active":"2024-02-10T05:43:43.834968+0000","last_peered":"2024-02-10T05:43:43.834968+0000","last_clean":"2024-02-10T05:43:43.834968+0000","last_became_active":"2024-02-10T05:43:17.847957+0000","last_became_peered":"2024-02-10T05:43:17.847957+0000","last_unstale":"2024-02-10T05:43:43.834968+0000","last_undegraded":"2024-02-10T05:43:43.834968+0000","last_fullsized":"2024-02-10T05:43:43.834968+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:59:22.013861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835385+0000","last_change":"2024-02-10T05:43:21.870341+0000","last_active":"2024-02-10T05:43:43.835385+0000","last_peered":"2024-02-10T05:43:43.835385+0000","last_clean":"2024-02-10T05:43:43.835385+0000","last_became_active":"2024-02-10T05:43:21.870050+0000","last_became_peered":"2024-02-10T05:43:21.870050+0000","last_unstale":"2024-02-10T05:43:43.835385+0000","last_undegraded":"2024-02-10T05:43:43.835385+0000","last_fullsized":"2024-02-10T05:43:43.835385+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:01:23.584030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.f","version":"23'2","reported_seq":39,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836966+0000","last_change":"2024-02-10T05:43:15.833202+0000","last_active":"2024-02-10T05:43:43.836966+0000","last_peered":"2024-02-10T05:43:43.836966+0000","last_clean":"2024-02-10T05:43:43.836966+0000","last_became_active":"2024-02-10T05:43:15.832908+0000","last_became_peered":"2024-02-10T05:43:15.832908+0000","last_unstale":"2024-02-10T05:43:43.836966+0000","last_undegraded":"2024-02-10T05:43:43.836966+0000","last_fullsized":"2024-02-10T05:43:43.836966+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:59:22.599607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.e","version":"30'17","reported_seq":54,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837235+0000","last_change":"2024-02-10T05:43:17.848205+0000","last_active":"2024-02-10T05:43:43.837235+0000","last_peered":"2024-02-10T05:43:43.837235+0000","last_clean":"2024-02-10T05:43:43.837235+0000","last_became_active":"2024-02-10T05:43:17.847892+0000","last_became_peered":"2024-02-10T05:43:17.847892+0000","last_unstale":"2024-02-10T05:43:43.837235+0000","last_undegraded":"2024-02-10T05:43:43.837235+0000","last_fullsized":"2024-02-10T05:43:43.837235+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:16:22.430428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882459+0000","last_change":"2024-02-10T05:43:19.860719+0000","last_active":"2024-02-10T05:43:55.882459+0000","last_peered":"2024-02-10T05:43:55.882459+0000","last_clean":"2024-02-10T05:43:55.882459+0000","last_became_active":"2024-02-10T05:43:19.860476+0000","last_became_peered":"2024-02-10T05:43:19.860476+0000","last_unstale":"2024-02-10T05:43:55.882459+0000","last_undegraded":"2024-02-10T05:43:55.882459+0000","last_fullsized":"2024-02-10T05:43:55.882459+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:02:08.541506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832988+0000","last_change":"2024-02-10T05:43:21.869808+0000","last_active":"2024-02-10T05:43:43.832988+0000","last_peered":"2024-02-10T05:43:43.832988+0000","last_clean":"2024-02-10T05:43:43.832988+0000","last_became_active":"2024-02-10T05:43:21.869472+0000","last_became_peered":"2024-02-10T05:43:21.869472+0000","last_unstale":"2024-02-10T05:43:43.832988+0000","last_undegraded":"2024-02-10T05:43:43.832988+0000","last_fullsized":"2024-02-10T05:43:43.832988+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:51:34.890026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835653+0000","last_change":"2024-02-10T05:43:15.839144+0000","last_active":"2024-02-10T05:43:43.835653+0000","last_peered":"2024-02-10T05:43:43.835653+0000","last_clean":"2024-02-10T05:43:43.835653+0000","last_became_active":"2024-02-10T05:43:15.838774+0000","last_became_peered":"2024-02-10T05:43:15.838774+0000","last_unstale":"2024-02-10T05:43:43.835653+0000","last_undegraded":"2024-02-10T05:43:43.835653+0000","last_fullsized":"2024-02-10T05:43:43.835653+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:30:21.148223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"30'16","reported_seq":55,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882405+0000","last_change":"2024-02-10T05:43:17.848010+0000","last_active":"2024-02-10T05:43:55.882405+0000","last_peered":"2024-02-10T05:43:55.882405+0000","last_clean":"2024-02-10T05:43:55.882405+0000","last_became_active":"2024-02-10T05:43:17.847533+0000","last_became_peered":"2024-02-10T05:43:17.847533+0000","last_unstale":"2024-02-10T05:43:55.882405+0000","last_undegraded":"2024-02-10T05:43:55.882405+0000","last_fullsized":"2024-02-10T05:43:55.882405+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:50:06.009222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834635+0000","last_change":"2024-02-10T05:43:19.860420+0000","last_active":"2024-02-10T05:43:43.834635+0000","last_peered":"2024-02-10T05:43:43.834635+0000","last_clean":"2024-02-10T05:43:43.834635+0000","last_became_active":"2024-02-10T05:43:19.859621+0000","last_became_peered":"2024-02-10T05:43:19.859621+0000","last_unstale":"2024-02-10T05:43:43.834635+0000","last_undegraded":"2024-02-10T05:43:43.834635+0000","last_fullsized":"2024-02-10T05:43:43.834635+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:13:50.841249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833158+0000","last_change":"2024-02-10T05:43:21.873625+0000","last_active":"2024-02-10T05:43:43.833158+0000","last_peered":"2024-02-10T05:43:43.833158+0000","last_clean":"2024-02-10T05:43:43.833158+0000","last_became_active":"2024-02-10T05:43:21.873199+0000","last_became_peered":"2024-02-10T05:43:21.873199+0000","last_unstale":"2024-02-10T05:43:43.833158+0000","last_undegraded":"2024-02-10T05:43:43.833158+0000","last_fullsized":"2024-02-10T05:43:43.833158+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:12:06.568634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838101+0000","last_change":"2024-02-10T05:43:15.836200+0000","last_active":"2024-02-10T05:43:43.838101+0000","last_peered":"2024-02-10T05:43:43.838101+0000","last_clean":"2024-02-10T05:43:43.838101+0000","last_became_active":"2024-02-10T05:43:15.835429+0000","last_became_peered":"2024-02-10T05:43:15.835429+0000","last_unstale":"2024-02-10T05:43:43.838101+0000","last_undegraded":"2024-02-10T05:43:43.838101+0000","last_fullsized":"2024-02-10T05:43:43.838101+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:14:11.657449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"30'25","reported_seq":71,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839246+0000","last_change":"2024-02-10T05:43:17.847802+0000","last_active":"2024-02-10T05:43:43.839246+0000","last_peered":"2024-02-10T05:43:43.839246+0000","last_clean":"2024-02-10T05:43:43.839246+0000","last_became_active":"2024-02-10T05:43:17.847498+0000","last_became_peered":"2024-02-10T05:43:17.847498+0000","last_unstale":"2024-02-10T05:43:43.839246+0000","last_undegraded":"2024-02-10T05:43:43.839246+0000","last_fullsized":"2024-02-10T05:43:43.839246+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:41:45.330257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833911+0000","last_change":"2024-02-10T05:43:19.863110+0000","last_active":"2024-02-10T05:43:43.833911+0000","last_peered":"2024-02-10T05:43:43.833911+0000","last_clean":"2024-02-10T05:43:43.833911+0000","last_became_active":"2024-02-10T05:43:19.862480+0000","last_became_peered":"2024-02-10T05:43:19.862480+0000","last_unstale":"2024-02-10T05:43:43.833911+0000","last_undegraded":"2024-02-10T05:43:43.833911+0000","last_fullsized":"2024-02-10T05:43:43.833911+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:25.669292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836302+0000","last_change":"2024-02-10T05:43:21.874425+0000","last_active":"2024-02-10T05:43:43.836302+0000","last_peered":"2024-02-10T05:43:43.836302+0000","last_clean":"2024-02-10T05:43:43.836302+0000","last_became_active":"2024-02-10T05:43:21.874178+0000","last_became_peered":"2024-02-10T05:43:21.874178+0000","last_unstale":"2024-02-10T05:43:43.836302+0000","last_undegraded":"2024-02-10T05:43:43.836302+0000","last_fullsized":"2024-02-10T05:43:43.836302+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:36:34.050976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.0","version":"30'30","reported_seq":79,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834105+0000","last_change":"2024-02-10T05:43:17.847871+0000","last_active":"2024-02-10T05:43:43.834105+0000","last_peered":"2024-02-10T05:43:43.834105+0000","last_clean":"2024-02-10T05:43:43.834105+0000","last_became_active":"2024-02-10T05:43:17.847584+0000","last_became_peered":"2024-02-10T05:43:17.847584+0000","last_unstale":"2024-02-10T05:43:43.834105+0000","last_undegraded":"2024-02-10T05:43:43.834105+0000","last_fullsized":"2024-02-10T05:43:43.834105+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:29:36.934501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833034+0000","last_change":"2024-02-10T05:43:15.832741+0000","last_active":"2024-02-10T05:43:43.833034+0000","last_peered":"2024-02-10T05:43:43.833034+0000","last_clean":"2024-02-10T05:43:43.833034+0000","last_became_active":"2024-02-10T05:43:15.832480+0000","last_became_peered":"2024-02-10T05:43:15.832480+0000","last_unstale":"2024-02-10T05:43:43.833034+0000","last_undegraded":"2024-02-10T05:43:43.833034+0000","last_fullsized":"2024-02-10T05:43:43.833034+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:34:56.042494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834541+0000","last_change":"2024-02-10T05:43:19.866739+0000","last_active":"2024-02-10T05:43:43.834541+0000","last_peered":"2024-02-10T05:43:43.834541+0000","last_clean":"2024-02-10T05:43:43.834541+0000","last_became_active":"2024-02-10T05:43:19.866422+0000","last_became_peered":"2024-02-10T05:43:19.866422+0000","last_unstale":"2024-02-10T05:43:43.834541+0000","last_undegraded":"2024-02-10T05:43:43.834541+0000","last_fullsized":"2024-02-10T05:43:43.834541+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:02:47.481107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835472+0000","last_change":"2024-02-10T05:43:21.870176+0000","last_active":"2024-02-10T05:43:43.835472+0000","last_peered":"2024-02-10T05:43:43.835472+0000","last_clean":"2024-02-10T05:43:43.835472+0000","last_became_active":"2024-02-10T05:43:21.869872+0000","last_became_peered":"2024-02-10T05:43:21.869872+0000","last_unstale":"2024-02-10T05:43:43.835472+0000","last_undegraded":"2024-02-10T05:43:43.835472+0000","last_fullsized":"2024-02-10T05:43:43.835472+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:31:57.366906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834231+0000","last_change":"2024-02-10T05:43:17.847724+0000","last_active":"2024-02-10T05:43:43.834231+0000","last_peered":"2024-02-10T05:43:43.834231+0000","last_clean":"2024-02-10T05:43:43.834231+0000","last_became_active":"2024-02-10T05:43:17.847166+0000","last_became_peered":"2024-02-10T05:43:17.847166+0000","last_unstale":"2024-02-10T05:43:43.834231+0000","last_undegraded":"2024-02-10T05:43:43.834231+0000","last_fullsized":"2024-02-10T05:43:43.834231+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:20.119597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838741+0000","last_change":"2024-02-10T05:43:15.836528+0000","last_active":"2024-02-10T05:43:43.838741+0000","last_peered":"2024-02-10T05:43:43.838741+0000","last_clean":"2024-02-10T05:43:43.838741+0000","last_became_active":"2024-02-10T05:43:15.835877+0000","last_became_peered":"2024-02-10T05:43:15.835877+0000","last_unstale":"2024-02-10T05:43:43.838741+0000","last_undegraded":"2024-02-10T05:43:43.838741+0000","last_fullsized":"2024-02-10T05:43:43.838741+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:37:59.156888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"31'208","reported_seq":314,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:45.930772+0000","last_change":"2024-02-10T05:43:03.910671+0000","last_active":"2024-02-10T05:43:45.930772+0000","last_peered":"2024-02-10T05:43:45.930772+0000","last_clean":"2024-02-10T05:43:45.930772+0000","last_became_active":"2024-02-10T05:43:03.910326+0000","last_became_peered":"2024-02-10T05:43:03.910326+0000","last_unstale":"2024-02-10T05:43:45.930772+0000","last_undegraded":"2024-02-10T05:43:45.930772+0000","last_fullsized":"2024-02-10T05:43:45.930772+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:02.902254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:02.902254+0000","last_clean_scrub_stamp":"2024-02-10T05:43:02.902254+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:57:11.322348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839686+0000","last_change":"2024-02-10T05:43:19.864376+0000","last_active":"2024-02-10T05:43:43.839686+0000","last_peered":"2024-02-10T05:43:43.839686+0000","last_clean":"2024-02-10T05:43:43.839686+0000","last_became_active":"2024-02-10T05:43:19.863867+0000","last_became_peered":"2024-02-10T05:43:19.863867+0000","last_unstale":"2024-02-10T05:43:43.839686+0000","last_undegraded":"2024-02-10T05:43:43.839686+0000","last_fullsized":"2024-02-10T05:43:43.839686+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:05:04.553384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839688+0000","last_change":"2024-02-10T05:43:21.875731+0000","last_active":"2024-02-10T05:43:43.839688+0000","last_peered":"2024-02-10T05:43:43.839688+0000","last_clean":"2024-02-10T05:43:43.839688+0000","last_became_active":"2024-02-10T05:43:21.874946+0000","last_became_peered":"2024-02-10T05:43:21.874946+0000","last_unstale":"2024-02-10T05:43:43.839688+0000","last_undegraded":"2024-02-10T05:43:43.839688+0000","last_fullsized":"2024-02-10T05:43:43.839688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:34:52.991754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"30'20","reported_seq":56,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833314+0000","last_change":"2024-02-10T05:43:17.851641+0000","last_active":"2024-02-10T05:43:43.833314+0000","last_peered":"2024-02-10T05:43:43.833314+0000","last_clean":"2024-02-10T05:43:43.833314+0000","last_became_active":"2024-02-10T05:43:17.851257+0000","last_became_peered":"2024-02-10T05:43:17.851257+0000","last_unstale":"2024-02-10T05:43:43.833314+0000","last_undegraded":"2024-02-10T05:43:43.833314+0000","last_fullsized":"2024-02-10T05:43:43.833314+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:44:17.022099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837162+0000","last_change":"2024-02-10T05:43:15.837997+0000","last_active":"2024-02-10T05:43:43.837162+0000","last_peered":"2024-02-10T05:43:43.837162+0000","last_clean":"2024-02-10T05:43:43.837162+0000","last_became_active":"2024-02-10T05:43:15.837364+0000","last_became_peered":"2024-02-10T05:43:15.837364+0000","last_unstale":"2024-02-10T05:43:43.837162+0000","last_undegraded":"2024-02-10T05:43:43.837162+0000","last_fullsized":"2024-02-10T05:43:43.837162+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T14:38:22.913893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836690+0000","last_change":"2024-02-10T05:43:19.863341+0000","last_active":"2024-02-10T05:43:43.836690+0000","last_peered":"2024-02-10T05:43:43.836690+0000","last_clean":"2024-02-10T05:43:43.836690+0000","last_became_active":"2024-02-10T05:43:19.863055+0000","last_became_peered":"2024-02-10T05:43:19.863055+0000","last_unstale":"2024-02-10T05:43:43.836690+0000","last_undegraded":"2024-02-10T05:43:43.836690+0000","last_fullsized":"2024-02-10T05:43:43.836690+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:00:12.233736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837797+0000","last_change":"2024-02-10T05:43:21.875911+0000","last_active":"2024-02-10T05:43:43.837797+0000","last_peered":"2024-02-10T05:43:43.837797+0000","last_clean":"2024-02-10T05:43:43.837797+0000","last_became_active":"2024-02-10T05:43:21.875027+0000","last_became_peered":"2024-02-10T05:43:21.875027+0000","last_unstale":"2024-02-10T05:43:43.837797+0000","last_undegraded":"2024-02-10T05:43:43.837797+0000","last_fullsized":"2024-02-10T05:43:43.837797+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:27:02.546201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836391+0000","last_change":"2024-02-10T05:43:15.839090+0000","last_active":"2024-02-10T05:43:43.836391+0000","last_peered":"2024-02-10T05:43:43.836391+0000","last_clean":"2024-02-10T05:43:43.836391+0000","last_became_active":"2024-02-10T05:43:15.838668+0000","last_became_peered":"2024-02-10T05:43:15.838668+0000","last_unstale":"2024-02-10T05:43:43.836391+0000","last_undegraded":"2024-02-10T05:43:43.836391+0000","last_fullsized":"2024-02-10T05:43:43.836391+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:14:42.909340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.4","version":"30'34","reported_seq":93,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882405+0000","last_change":"2024-02-10T05:43:17.853328+0000","last_active":"2024-02-10T05:43:55.882405+0000","last_peered":"2024-02-10T05:43:55.882405+0000","last_clean":"2024-02-10T05:43:55.882405+0000","last_became_active":"2024-02-10T05:43:17.852388+0000","last_became_peered":"2024-02-10T05:43:17.852388+0000","last_unstale":"2024-02-10T05:43:55.882405+0000","last_undegraded":"2024-02-10T05:43:55.882405+0000","last_fullsized":"2024-02-10T05:43:55.882405+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:34:37.717866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882416+0000","last_change":"2024-02-10T05:43:19.860396+0000","last_active":"2024-02-10T05:43:55.882416+0000","last_peered":"2024-02-10T05:43:55.882416+0000","last_clean":"2024-02-10T05:43:55.882416+0000","last_became_active":"2024-02-10T05:43:19.859572+0000","last_became_peered":"2024-02-10T05:43:19.859572+0000","last_unstale":"2024-02-10T05:43:55.882416+0000","last_undegraded":"2024-02-10T05:43:55.882416+0000","last_fullsized":"2024-02-10T05:43:55.882416+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:06:57.757481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837199+0000","last_change":"2024-02-10T05:43:21.875794+0000","last_active":"2024-02-10T05:43:43.837199+0000","last_peered":"2024-02-10T05:43:43.837199+0000","last_clean":"2024-02-10T05:43:43.837199+0000","last_became_active":"2024-02-10T05:43:21.875065+0000","last_became_peered":"2024-02-10T05:43:21.875065+0000","last_unstale":"2024-02-10T05:43:43.837199+0000","last_undegraded":"2024-02-10T05:43:43.837199+0000","last_fullsized":"2024-02-10T05:43:43.837199+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:57:18.944049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":28,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836302+0000","last_change":"2024-02-10T05:43:15.832744+0000","last_active":"2024-02-10T05:43:43.836302+0000","last_peered":"2024-02-10T05:43:43.836302+0000","last_clean":"2024-02-10T05:43:43.836302+0000","last_became_active":"2024-02-10T05:43:15.832523+0000","last_became_peered":"2024-02-10T05:43:15.832523+0000","last_unstale":"2024-02-10T05:43:43.836302+0000","last_undegraded":"2024-02-10T05:43:43.836302+0000","last_fullsized":"2024-02-10T05:43:43.836302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:28:33.379519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"30'21","reported_seq":64,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835951+0000","last_change":"2024-02-10T05:43:17.851969+0000","last_active":"2024-02-10T05:43:43.835951+0000","last_peered":"2024-02-10T05:43:43.835951+0000","last_clean":"2024-02-10T05:43:43.835951+0000","last_became_active":"2024-02-10T05:43:17.851660+0000","last_became_peered":"2024-02-10T05:43:17.851660+0000","last_unstale":"2024-02-10T05:43:43.835951+0000","last_undegraded":"2024-02-10T05:43:43.835951+0000","last_fullsized":"2024-02-10T05:43:43.835951+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T15:57:27.504325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882384+0000","last_change":"2024-02-10T05:43:19.863492+0000","last_active":"2024-02-10T05:43:55.882384+0000","last_peered":"2024-02-10T05:43:55.882384+0000","last_clean":"2024-02-10T05:43:55.882384+0000","last_became_active":"2024-02-10T05:43:19.863149+0000","last_became_peered":"2024-02-10T05:43:19.863149+0000","last_unstale":"2024-02-10T05:43:55.882384+0000","last_undegraded":"2024-02-10T05:43:55.882384+0000","last_fullsized":"2024-02-10T05:43:55.882384+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:59:28.219067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835349+0000","last_change":"2024-02-10T05:43:21.870225+0000","last_active":"2024-02-10T05:43:43.835349+0000","last_peered":"2024-02-10T05:43:43.835349+0000","last_clean":"2024-02-10T05:43:43.835349+0000","last_became_active":"2024-02-10T05:43:21.869930+0000","last_became_peered":"2024-02-10T05:43:21.869930+0000","last_unstale":"2024-02-10T05:43:43.835349+0000","last_undegraded":"2024-02-10T05:43:43.835349+0000","last_fullsized":"2024-02-10T05:43:43.835349+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:15:20.444051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840351+0000","last_change":"2024-02-10T05:43:15.836108+0000","last_active":"2024-02-10T05:43:43.840351+0000","last_peered":"2024-02-10T05:43:43.840351+0000","last_clean":"2024-02-10T05:43:43.840351+0000","last_became_active":"2024-02-10T05:43:15.835598+0000","last_became_peered":"2024-02-10T05:43:15.835598+0000","last_unstale":"2024-02-10T05:43:43.840351+0000","last_undegraded":"2024-02-10T05:43:43.840351+0000","last_fullsized":"2024-02-10T05:43:43.840351+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:02:07.036035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"30'12","reported_seq":41,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835941+0000","last_change":"2024-02-10T05:43:17.852433+0000","last_active":"2024-02-10T05:43:43.835941+0000","last_peered":"2024-02-10T05:43:43.835941+0000","last_clean":"2024-02-10T05:43:43.835941+0000","last_became_active":"2024-02-10T05:43:17.852124+0000","last_became_peered":"2024-02-10T05:43:17.852124+0000","last_unstale":"2024-02-10T05:43:43.835941+0000","last_undegraded":"2024-02-10T05:43:43.835941+0000","last_fullsized":"2024-02-10T05:43:43.835941+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:13:26.424699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834511+0000","last_change":"2024-02-10T05:43:19.863338+0000","last_active":"2024-02-10T05:43:43.834511+0000","last_peered":"2024-02-10T05:43:43.834511+0000","last_clean":"2024-02-10T05:43:43.834511+0000","last_became_active":"2024-02-10T05:43:19.863028+0000","last_became_peered":"2024-02-10T05:43:19.863028+0000","last_unstale":"2024-02-10T05:43:43.834511+0000","last_undegraded":"2024-02-10T05:43:43.834511+0000","last_fullsized":"2024-02-10T05:43:43.834511+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:34:50.860344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839596+0000","last_change":"2024-02-10T05:43:21.875993+0000","last_active":"2024-02-10T05:43:43.839596+0000","last_peered":"2024-02-10T05:43:43.839596+0000","last_clean":"2024-02-10T05:43:43.839596+0000","last_became_active":"2024-02-10T05:43:21.875468+0000","last_became_peered":"2024-02-10T05:43:21.875468+0000","last_unstale":"2024-02-10T05:43:43.839596+0000","last_undegraded":"2024-02-10T05:43:43.839596+0000","last_fullsized":"2024-02-10T05:43:43.839596+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:47:46.160440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882366+0000","last_change":"2024-02-10T05:43:19.864323+0000","last_active":"2024-02-10T05:43:55.882366+0000","last_peered":"2024-02-10T05:43:55.882366+0000","last_clean":"2024-02-10T05:43:55.882366+0000","last_became_active":"2024-02-10T05:43:19.863883+0000","last_became_peered":"2024-02-10T05:43:19.863883+0000","last_unstale":"2024-02-10T05:43:55.882366+0000","last_undegraded":"2024-02-10T05:43:55.882366+0000","last_fullsized":"2024-02-10T05:43:55.882366+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:07:14.896925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837426+0000","last_change":"2024-02-10T05:43:15.837978+0000","last_active":"2024-02-10T05:43:43.837426+0000","last_peered":"2024-02-10T05:43:43.837426+0000","last_clean":"2024-02-10T05:43:43.837426+0000","last_became_active":"2024-02-10T05:43:15.836913+0000","last_became_peered":"2024-02-10T05:43:15.836913+0000","last_unstale":"2024-02-10T05:43:43.837426+0000","last_undegraded":"2024-02-10T05:43:43.837426+0000","last_fullsized":"2024-02-10T05:43:43.837426+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:21:11.634868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"30'24","reported_seq":70,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834812+0000","last_change":"2024-02-10T05:43:17.848295+0000","last_active":"2024-02-10T05:43:43.834812+0000","last_peered":"2024-02-10T05:43:43.834812+0000","last_clean":"2024-02-10T05:43:43.834812+0000","last_became_active":"2024-02-10T05:43:17.847924+0000","last_became_peered":"2024-02-10T05:43:17.847924+0000","last_unstale":"2024-02-10T05:43:43.834812+0000","last_undegraded":"2024-02-10T05:43:43.834812+0000","last_fullsized":"2024-02-10T05:43:43.834812+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:06:22.286915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836781+0000","last_change":"2024-02-10T05:43:21.870347+0000","last_active":"2024-02-10T05:43:43.836781+0000","last_peered":"2024-02-10T05:43:43.836781+0000","last_clean":"2024-02-10T05:43:43.836781+0000","last_became_active":"2024-02-10T05:43:21.869996+0000","last_became_peered":"2024-02-10T05:43:21.869996+0000","last_unstale":"2024-02-10T05:43:43.836781+0000","last_undegraded":"2024-02-10T05:43:43.836781+0000","last_fullsized":"2024-02-10T05:43:43.836781+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:40:59.131735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834322+0000","last_change":"2024-02-10T05:43:19.864326+0000","last_active":"2024-02-10T05:43:43.834322+0000","last_peered":"2024-02-10T05:43:43.834322+0000","last_clean":"2024-02-10T05:43:43.834322+0000","last_became_active":"2024-02-10T05:43:19.863987+0000","last_became_peered":"2024-02-10T05:43:19.863987+0000","last_unstale":"2024-02-10T05:43:43.834322+0000","last_undegraded":"2024-02-10T05:43:43.834322+0000","last_fullsized":"2024-02-10T05:43:43.834322+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:41:23.518744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832862+0000","last_change":"2024-02-10T05:43:15.840029+0000","last_active":"2024-02-10T05:43:43.832862+0000","last_peered":"2024-02-10T05:43:43.832862+0000","last_clean":"2024-02-10T05:43:43.832862+0000","last_became_active":"2024-02-10T05:43:15.839596+0000","last_became_peered":"2024-02-10T05:43:15.839596+0000","last_unstale":"2024-02-10T05:43:43.832862+0000","last_undegraded":"2024-02-10T05:43:43.832862+0000","last_fullsized":"2024-02-10T05:43:43.832862+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:28.254239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"30'27","reported_seq":72,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834608+0000","last_change":"2024-02-10T05:43:17.847867+0000","last_active":"2024-02-10T05:43:43.834608+0000","last_peered":"2024-02-10T05:43:43.834608+0000","last_clean":"2024-02-10T05:43:43.834608+0000","last_became_active":"2024-02-10T05:43:17.847582+0000","last_became_peered":"2024-02-10T05:43:17.847582+0000","last_unstale":"2024-02-10T05:43:43.834608+0000","last_undegraded":"2024-02-10T05:43:43.834608+0000","last_fullsized":"2024-02-10T05:43:43.834608+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:16:06.397863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835802+0000","last_change":"2024-02-10T05:43:21.868219+0000","last_active":"2024-02-10T05:43:43.835802+0000","last_peered":"2024-02-10T05:43:43.835802+0000","last_clean":"2024-02-10T05:43:43.835802+0000","last_became_active":"2024-02-10T05:43:21.867956+0000","last_became_peered":"2024-02-10T05:43:21.867956+0000","last_unstale":"2024-02-10T05:43:43.835802+0000","last_undegraded":"2024-02-10T05:43:43.835802+0000","last_fullsized":"2024-02-10T05:43:43.835802+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T13:06:01.917026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835190+0000","last_change":"2024-02-10T05:43:15.833330+0000","last_active":"2024-02-10T05:43:43.835190+0000","last_peered":"2024-02-10T05:43:43.835190+0000","last_clean":"2024-02-10T05:43:43.835190+0000","last_became_active":"2024-02-10T05:43:15.833038+0000","last_became_peered":"2024-02-10T05:43:15.833038+0000","last_unstale":"2024-02-10T05:43:43.835190+0000","last_undegraded":"2024-02-10T05:43:43.835190+0000","last_fullsized":"2024-02-10T05:43:43.835190+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:56:10.612538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"30'15","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834150+0000","last_change":"2024-02-10T05:43:17.848286+0000","last_active":"2024-02-10T05:43:43.834150+0000","last_peered":"2024-02-10T05:43:43.834150+0000","last_clean":"2024-02-10T05:43:43.834150+0000","last_became_active":"2024-02-10T05:43:17.847826+0000","last_became_peered":"2024-02-10T05:43:17.847826+0000","last_unstale":"2024-02-10T05:43:43.834150+0000","last_undegraded":"2024-02-10T05:43:43.834150+0000","last_fullsized":"2024-02-10T05:43:43.834150+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:39:27.311005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838835+0000","last_change":"2024-02-10T05:43:19.863466+0000","last_active":"2024-02-10T05:43:43.838835+0000","last_peered":"2024-02-10T05:43:43.838835+0000","last_clean":"2024-02-10T05:43:43.838835+0000","last_became_active":"2024-02-10T05:43:19.863094+0000","last_became_peered":"2024-02-10T05:43:19.863094+0000","last_unstale":"2024-02-10T05:43:43.838835+0000","last_undegraded":"2024-02-10T05:43:43.838835+0000","last_fullsized":"2024-02-10T05:43:43.838835+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:39:32.002402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835489+0000","last_change":"2024-02-10T05:43:21.870233+0000","last_active":"2024-02-10T05:43:43.835489+0000","last_peered":"2024-02-10T05:43:43.835489+0000","last_clean":"2024-02-10T05:43:43.835489+0000","last_became_active":"2024-02-10T05:43:21.869926+0000","last_became_peered":"2024-02-10T05:43:21.869926+0000","last_unstale":"2024-02-10T05:43:43.835489+0000","last_undegraded":"2024-02-10T05:43:43.835489+0000","last_fullsized":"2024-02-10T05:43:43.835489+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:22:39.569819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"30'8","reported_seq":37,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:55.882364+0000","last_change":"2024-02-10T05:43:19.863670+0000","last_active":"2024-02-10T05:43:55.882364+0000","last_peered":"2024-02-10T05:43:55.882364+0000","last_clean":"2024-02-10T05:43:55.882364+0000","last_became_active":"2024-02-10T05:43:19.863344+0000","last_became_peered":"2024-02-10T05:43:19.863344+0000","last_unstale":"2024-02-10T05:43:55.882364+0000","last_undegraded":"2024-02-10T05:43:55.882364+0000","last_fullsized":"2024-02-10T05:43:55.882364+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:16:44.206305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835559+0000","last_change":"2024-02-10T05:43:15.839259+0000","last_active":"2024-02-10T05:43:43.835559+0000","last_peered":"2024-02-10T05:43:43.835559+0000","last_clean":"2024-02-10T05:43:43.835559+0000","last_became_active":"2024-02-10T05:43:15.838942+0000","last_became_peered":"2024-02-10T05:43:15.838942+0000","last_unstale":"2024-02-10T05:43:43.835559+0000","last_undegraded":"2024-02-10T05:43:43.835559+0000","last_fullsized":"2024-02-10T05:43:43.835559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:10:58.449047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"30'25","reported_seq":66,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.838298+0000","last_change":"2024-02-10T05:43:17.849835+0000","last_active":"2024-02-10T05:43:43.838298+0000","last_peered":"2024-02-10T05:43:43.838298+0000","last_clean":"2024-02-10T05:43:43.838298+0000","last_became_active":"2024-02-10T05:43:17.849540+0000","last_became_peered":"2024-02-10T05:43:17.849540+0000","last_unstale":"2024-02-10T05:43:43.838298+0000","last_undegraded":"2024-02-10T05:43:43.838298+0000","last_fullsized":"2024-02-10T05:43:43.838298+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:05:46.073612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.834732+0000","last_change":"2024-02-10T05:43:21.868264+0000","last_active":"2024-02-10T05:43:43.834732+0000","last_peered":"2024-02-10T05:43:43.834732+0000","last_clean":"2024-02-10T05:43:43.834732+0000","last_became_active":"2024-02-10T05:43:21.867985+0000","last_became_peered":"2024-02-10T05:43:21.867985+0000","last_unstale":"2024-02-10T05:43:43.834732+0000","last_undegraded":"2024-02-10T05:43:43.834732+0000","last_fullsized":"2024-02-10T05:43:43.834732+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:23.863227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836156+0000","last_change":"2024-02-10T05:43:15.834373+0000","last_active":"2024-02-10T05:43:43.836156+0000","last_peered":"2024-02-10T05:43:43.836156+0000","last_clean":"2024-02-10T05:43:43.836156+0000","last_became_active":"2024-02-10T05:43:15.834049+0000","last_became_peered":"2024-02-10T05:43:15.834049+0000","last_unstale":"2024-02-10T05:43:43.836156+0000","last_undegraded":"2024-02-10T05:43:43.836156+0000","last_fullsized":"2024-02-10T05:43:43.836156+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T11:59:30.697677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"30'17","reported_seq":51,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840413+0000","last_change":"2024-02-10T05:43:17.848298+0000","last_active":"2024-02-10T05:43:43.840413+0000","last_peered":"2024-02-10T05:43:43.840413+0000","last_clean":"2024-02-10T05:43:43.840413+0000","last_became_active":"2024-02-10T05:43:17.847854+0000","last_became_peered":"2024-02-10T05:43:17.847854+0000","last_unstale":"2024-02-10T05:43:43.840413+0000","last_undegraded":"2024-02-10T05:43:43.840413+0000","last_fullsized":"2024-02-10T05:43:43.840413+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T17:12:12.480845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837625+0000","last_change":"2024-02-10T05:43:19.863557+0000","last_active":"2024-02-10T05:43:43.837625+0000","last_peered":"2024-02-10T05:43:43.837625+0000","last_clean":"2024-02-10T05:43:43.837625+0000","last_became_active":"2024-02-10T05:43:19.863269+0000","last_became_peered":"2024-02-10T05:43:19.863269+0000","last_unstale":"2024-02-10T05:43:43.837625+0000","last_undegraded":"2024-02-10T05:43:43.837625+0000","last_fullsized":"2024-02-10T05:43:43.837625+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:08:22.931566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.832726+0000","last_change":"2024-02-10T05:43:21.869921+0000","last_active":"2024-02-10T05:43:43.832726+0000","last_peered":"2024-02-10T05:43:43.832726+0000","last_clean":"2024-02-10T05:43:43.832726+0000","last_became_active":"2024-02-10T05:43:21.869608+0000","last_became_peered":"2024-02-10T05:43:21.869608+0000","last_unstale":"2024-02-10T05:43:43.832726+0000","last_undegraded":"2024-02-10T05:43:43.832726+0000","last_fullsized":"2024-02-10T05:43:43.832726+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T12:28:23.857899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836247+0000","last_change":"2024-02-10T05:43:15.839471+0000","last_active":"2024-02-10T05:43:43.836247+0000","last_peered":"2024-02-10T05:43:43.836247+0000","last_clean":"2024-02-10T05:43:43.836247+0000","last_became_active":"2024-02-10T05:43:15.838801+0000","last_became_peered":"2024-02-10T05:43:15.838801+0000","last_unstale":"2024-02-10T05:43:43.836247+0000","last_undegraded":"2024-02-10T05:43:43.836247+0000","last_fullsized":"2024-02-10T05:43:43.836247+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T16:54:08.853699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"30'10","reported_seq":38,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.839103+0000","last_change":"2024-02-10T05:43:17.848303+0000","last_active":"2024-02-10T05:43:43.839103+0000","last_peered":"2024-02-10T05:43:43.839103+0000","last_clean":"2024-02-10T05:43:43.839103+0000","last_became_active":"2024-02-10T05:43:17.847968+0000","last_became_peered":"2024-02-10T05:43:17.847968+0000","last_unstale":"2024-02-10T05:43:43.839103+0000","last_undegraded":"2024-02-10T05:43:43.839103+0000","last_fullsized":"2024-02-10T05:43:43.839103+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:14:30.146762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.833093+0000","last_change":"2024-02-10T05:43:19.862894+0000","last_active":"2024-02-10T05:43:43.833093+0000","last_peered":"2024-02-10T05:43:43.833093+0000","last_clean":"2024-02-10T05:43:43.833093+0000","last_became_active":"2024-02-10T05:43:19.862344+0000","last_became_peered":"2024-02-10T05:43:19.862344+0000","last_unstale":"2024-02-10T05:43:43.833093+0000","last_undegraded":"2024-02-10T05:43:43.833093+0000","last_fullsized":"2024-02-10T05:43:43.833093+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:52:16.872156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":15,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.837262+0000","last_change":"2024-02-10T05:43:21.871037+0000","last_active":"2024-02-10T05:43:43.837262+0000","last_peered":"2024-02-10T05:43:43.837262+0000","last_clean":"2024-02-10T05:43:43.837262+0000","last_became_active":"2024-02-10T05:43:21.870074+0000","last_became_peered":"2024-02-10T05:43:21.870074+0000","last_unstale":"2024-02-10T05:43:43.837262+0000","last_undegraded":"2024-02-10T05:43:43.837262+0000","last_fullsized":"2024-02-10T05:43:43.837262+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:20.852291+0000","last_clean_scrub_stamp":"2024-02-10T05:43:20.852291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:49:37.727636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":27,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.836084+0000","last_change":"2024-02-10T05:43:15.834089+0000","last_active":"2024-02-10T05:43:43.836084+0000","last_peered":"2024-02-10T05:43:43.836084+0000","last_clean":"2024-02-10T05:43:43.836084+0000","last_became_active":"2024-02-10T05:43:15.833752+0000","last_became_peered":"2024-02-10T05:43:15.833752+0000","last_unstale":"2024-02-10T05:43:43.836084+0000","last_undegraded":"2024-02-10T05:43:43.836084+0000","last_fullsized":"2024-02-10T05:43:43.836084+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:14.817525+0000","last_clean_scrub_stamp":"2024-02-10T05:43:14.817525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:43:14.753366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"30'15","reported_seq":48,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.840484+0000","last_change":"2024-02-10T05:43:17.848508+0000","last_active":"2024-02-10T05:43:43.840484+0000","last_peered":"2024-02-10T05:43:43.840484+0000","last_clean":"2024-02-10T05:43:43.840484+0000","last_became_active":"2024-02-10T05:43:17.848160+0000","last_became_peered":"2024-02-10T05:43:17.848160+0000","last_unstale":"2024-02-10T05:43:43.840484+0000","last_undegraded":"2024-02-10T05:43:43.840484+0000","last_fullsized":"2024-02-10T05:43:43.840484+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":24,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:16.830400+0000","last_clean_scrub_stamp":"2024-02-10T05:43:16.830400+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:26.986686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":19,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-02-10T05:43:43.835104+0000","last_change":"2024-02-10T05:43:19.860956+0000","last_active":"2024-02-10T05:43:43.835104+0000","last_peered":"2024-02-10T05:43:43.835104+0000","last_clean":"2024-02-10T05:43:43.835104+0000","last_became_active":"2024-02-10T05:43:19.860684+0000","last_became_peered":"2024-02-10T05:43:19.860684+0000","last_unstale":"2024-02-10T05:43:43.835104+0000","last_undegraded":"2024-02-10T05:43:43.835104+0000","last_fullsized":"2024-02-10T05:43:43.835104+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":26,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-10T05:43:18.843195+0000","last_clean_scrub_stamp":"2024-02-10T05:43:18.843195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:22:54.390879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":3},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":588,"ondisk_log_size":588,"up":96,"acting":96,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":252,"num_read_kb":342,"num_write":267,"num_write_kb":4504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":208,"ondisk_log_size":208,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1456,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":18,"seq":77309411343,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704688,"statfs":{"total":95990841344,"available":95953600512,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:44:03 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.38},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.33300000000000002}]},{"osd":1,"last update":"Sat Feb 10 05:44:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":1.827,"5min":1.827,"15min":1.827},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.44700000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738393,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93704688,"statfs":{"total":95990841344,"available":95953600512,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Feb 10 05:43:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.45100000000000001}]},{"osd":2,"last update":"Sat Feb 10 05:44:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.51600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":129,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93741056,"kb_used":32208,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93708848,"statfs":{"total":95990841344,"available":95957860352,"internally_reserved":0,"allocated":1052672,"data_stored":700115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Feb 10 05:43:40 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":3702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-10T05:44:08.192 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-10T05:44:08.192 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-10T05:44:08.192 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-10T05:44:08.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-10T05:44:08.802 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 podman[54361]: 2024-02-10 05:44:08.342607038 +0000 UTC m=+0.017787784 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-02-10T05:44:08.802 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 podman[54361]: 2024-02-10 05:44:08.46238173 +0000 UTC m=+0.137562513 volume create 38ab937cb7d3cefbac50d9ed0eaa9032e56bd6770132a4fca11310b4d93489ac 2024-02-10T05:44:08.802 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 podman[54361]: 2024-02-10 05:44:08.563118079 +0000 UTC m=+0.238298820 container create 9787d0a2b423d4cd723825af9b8c1581344e912b8e2ffe16b902095abe6c5454 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:44:09.190 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 podman[54361]: 2024-02-10 05:44:08.927520627 +0000 UTC m=+0.602701369 container init 9787d0a2b423d4cd723825af9b8c1581344e912b8e2ffe16b902095abe6c5454 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 podman[54361]: 2024-02-10 05:44:08.932933249 +0000 UTC m=+0.608113993 container start 9787d0a2b423d4cd723825af9b8c1581344e912b8e2ffe16b902095abe6c5454 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.945Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.945Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.947Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.29 port=9094 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.948Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.971Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.972Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.975Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-02-10T05:44:09.237 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:08.975Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-02-10T05:44:09.238 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:09 smithi029 bash[54361]: 9787d0a2b423d4cd723825af9b8c1581344e912b8e2ffe16b902095abe6c5454 2024-02-10T05:44:09.238 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:09 smithi029 systemd[1]: Started Ceph alertmanager.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:44:09.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:09 smithi110 ceph-mon[30869]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:09.489 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:09 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:09 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:09 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.490 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:09 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-mon[25692]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:09.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:09] ENGINE Bus STOPPING 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:09] ENGINE Bus STOPPED 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:09] ENGINE Bus STARTING 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:09] ENGINE Serving on http://:::9283 2024-02-10T05:44:09.491 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:09 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:09] ENGINE Bus STARTED 2024-02-10T05:44:09.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:09 smithi105 ceph-mon[29854]: pgmap v15: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:09.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:09 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:09 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:09 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:09.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:09 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-mon[25692]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:44:10.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:44:10.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-mon[25692]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:44:10.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-mon[25692]: Deploying daemon grafana.a on smithi105 2024-02-10T05:44:10.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:10 smithi105 ceph-mon[29854]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:44:10.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:10 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:10 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:10 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:44:10.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:10 smithi105 ceph-mon[29854]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:44:10.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:10 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:10 smithi105 ceph-mon[29854]: Deploying daemon grafana.a on smithi105 2024-02-10T05:44:10.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:10 smithi110 ceph-mon[30869]: Regenerating cephadm self-signed grafana TLS certificates 2024-02-10T05:44:10.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:10 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:10 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:10 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:44:10.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:10 smithi110 ceph-mon[30869]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-02-10T05:44:10.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:10 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:10.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:10 smithi110 ceph-mon[30869]: Deploying daemon grafana.a on smithi105 2024-02-10T05:44:11.106 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-02-10T05:44:11.107 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-10T05:44:11.107 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:10 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:10.948Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000867878s 2024-02-10T05:44:11.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:11 smithi029 ceph-mon[25692]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:11.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:11 smithi029 ceph-mon[25692]: from='client.? 172.21.15.29:0/2222798473' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:44:11.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:11 smithi105 ceph-mon[29854]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:11.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:11 smithi105 ceph-mon[29854]: from='client.? 172.21.15.29:0/2222798473' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:44:11.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:11 smithi110 ceph-mon[30869]: pgmap v16: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:11.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:11 smithi110 ceph-mon[30869]: from='client.? 172.21.15.29:0/2222798473' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-10T05:44:11.787 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-10T05:44:11.788 INFO:tasks.cephadm:Setup complete, yielding 2024-02-10T05:44:11.788 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-10T05:44:11.798 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi029.front.sepia.ceph.com 2024-02-10T05:44:11.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'stat -c '"'"'%u %g'"'"' /var/log/ceph | grep '"'"'167 167'"'"'' 2024-02-10T05:44:12.046 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:12.337 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:12 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ::ffff:172.21.15.105 - - [10/Feb/2024:05:44:12] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:44:13.544 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:13 smithi029 ceph-mon[25692]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:13.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:13 smithi105 ceph-mon[29854]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:13.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:13 smithi110 ceph-mon[30869]: pgmap v17: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:13.880 INFO:teuthology.orchestra.run.smithi029.stdout:167 167 2024-02-10T05:44:14.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-10T05:44:14.753 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:14 smithi029 ceph-mon[25692]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:14 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:15.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:14 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:15.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:14 smithi110 ceph-mon[30869]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:15.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:14 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:15.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:14 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:15.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:14 smithi105 ceph-mon[29854]: pgmap v18: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:15.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:14 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:15.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:14 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:16.781 INFO:teuthology.orchestra.run.smithi029.stdout:Backend: cephadm 2024-02-10T05:44:16.781 INFO:teuthology.orchestra.run.smithi029.stdout:Available: Yes 2024-02-10T05:44:16.781 INFO:teuthology.orchestra.run.smithi029.stdout:Paused: No 2024-02-10T05:44:17.211 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:16 smithi105 ceph-mon[29854]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:17.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:16 smithi110 ceph-mon[30869]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:17.384 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:16 smithi029 ceph-mon[25692]: pgmap v19: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:17.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-10T05:44:17.752 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:18.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:17 smithi029 ceph-mon[25692]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:18.231 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:17 smithi105 ceph-mon[29854]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:18.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:17 smithi110 ceph-mon[30869]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:19.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:18 smithi029 ceph-mon[25692]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:19.235 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:44:18 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:44:18.950Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002812562s 2024-02-10T05:44:19.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:18 smithi110 ceph-mon[30869]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:19.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:18 smithi105 ceph-mon[29854]: pgmap v20: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:19.819 INFO:teuthology.orchestra.run.smithi029.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-10T05:44:19.819 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager.a smithi029 *:9093,9094 starting - - - - 2024-02-10T05:44:19.819 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.x smithi105 *:8443,8765 running (2m) 33s ago 2m 447M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b53f4d6dc2ba 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.y smithi029 *:9283,8765 running (4m) 26s ago 4m 477M - 19.0.0-1361-g290212a1 c328ab9dcfc7 9698232a4e27 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:mgr.z smithi110 *:8443,8765 running (2m) 34s ago 2m 447M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ff4c8b64a0b2 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a smithi029 running (4m) 26s ago 4m 50.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 2013fc10e01e 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b smithi105 running (2m) 33s ago 2m 41.6M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 01e2d35d4f7a 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c smithi110 running (2m) 34s ago 2m 43.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 57ca5780fbb9 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 smithi029 running (2m) 26s ago 2m 59.0M 15.6G 19.0.0-1361-g290212a1 c328ab9dcfc7 7fead6174493 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 smithi105 running (107s) 33s ago 107s 60.0M 15.6G 19.0.0-1361-g290212a1 c328ab9dcfc7 de67ba3154ac 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 smithi110 running (83s) 34s ago 83s 60.1M 16.6G 19.0.0-1361-g290212a1 c328ab9dcfc7 8b31da414787 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus.a smithi105 *:9095 running (45s) 33s ago 44s 25.7M - 2.43.0 a07b618ecd1d 7741dbf83128 2024-02-10T05:44:19.820 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.a smithi029 *:80 running (65s) 26s ago 65s 79.7M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6b1c9238d089 2024-02-10T05:44:20.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-10T05:44:20.684 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:21.070 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:20 smithi029 ceph-mon[25692]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:21.070 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:20 smithi029 ceph-mon[25692]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:21.080 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:20 smithi105 ceph-mon[29854]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:21.080 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:20 smithi105 ceph-mon[29854]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:21.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:20 smithi110 ceph-mon[30869]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:21.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:20 smithi110 ceph-mon[30869]: pgmap v21: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:22.134 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:22 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ::ffff:172.21.15.105 - - [10/Feb/2024:05:44:22] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:44:22.590 INFO:teuthology.orchestra.run.smithi029.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:alertmanager ?:9093,9094 0/1 - 63s smithi029=a;count:1 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:grafana ?:3000 0/1 - 61s smithi105=a;count:1 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:mgr 3/3 37s ago 2m smithi029=y;smithi105=x;smithi110=z;count:3 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:mon 3/3 37s ago 3m smithi029:172.21.15.29=a;smithi105:172.21.15.105=b;smithi110:172.21.15.110=c;count:3 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:node-exporter ?:9100 0/3 - 66s smithi029=a;smithi105=b;smithi110=c;count:3 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:osd 3 37s ago - 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:prometheus ?:9095 1/1 36s ago 69s smithi105=a;count:1 2024-02-10T05:44:22.591 INFO:teuthology.orchestra.run.smithi029.stdout:rgw.foo.a ?:80 1/1 28s ago 68s smithi029=foo.a;count:1 2024-02-10T05:44:22.607 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 podman[42957]: 2024-02-10 05:44:22.502200889 +0000 UTC m=+0.136114908 container create ddfd594a50f1c6466d2d05685562dd8d1f32524c019746e838b84873056b69ae (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , name=ubi8, version=8.5, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.buildah.version=1.32.0, release=239.1651231664, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public) 2024-02-10T05:44:22.980 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-mon[29854]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:22.980 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-mon[29854]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:22.980 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:22.980 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:22.980 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:22.980 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:22.981 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 podman[42957]: 2024-02-10 05:44:22.730148214 +0000 UTC m=+0.364062199 container init ddfd594a50f1c6466d2d05685562dd8d1f32524c019746e838b84873056b69ae (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.buildah.version=1.32.0, maintainer=Paul Cuzner , distribution-scope=public, version=8.5, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.openshift.tags=base rhel8) 2024-02-10T05:44:22.981 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 podman[42957]: 2024-02-10 05:44:22.735492301 +0000 UTC m=+0.369406277 container start ddfd594a50f1c6466d2d05685562dd8d1f32524c019746e838b84873056b69ae (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a, distribution-scope=public, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, release=239.1651231664, description=Ceph Grafana Container, vcs-type=git, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, name=ubi8, io.openshift.expose-services=) 2024-02-10T05:44:22.981 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 bash[42957]: ddfd594a50f1c6466d2d05685562dd8d1f32524c019746e838b84873056b69ae 2024-02-10T05:44:22.981 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 systemd[1]: Started Ceph grafana.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:44:22.981 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891616801Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-02-10T05:44:22.981 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.8918196Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-02-10T05:44:22.981 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891838326Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891845871Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891851997Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891857898Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891863556Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891869684Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891875547Z level=info msg="Path Home" path=/usr/share/grafana 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.89188153Z level=info msg="Path Data" path=/var/lib/grafana 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.89188748Z level=info msg="Path Logs" path=/var/log/grafana 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.89189309Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891898896Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=settings t=2024-02-10T05:44:22.891904573Z level=info msg="App mode production" 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=sqlstore t=2024-02-10T05:44:22.891952399Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-02-10T05:44:22.982 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=sqlstore t=2024-02-10T05:44:22.891969223Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.89286344Z level=info msg="Starting DB migrations" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.892944063Z level=info msg="Executing migration" id="create migration_log table" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.895671287Z level=info msg="Executing migration" id="create user table" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.898217384Z level=info msg="Executing migration" id="add unique index user.login" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.898698168Z level=info msg="Executing migration" id="add unique index user.email" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.899155479Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.899743712Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.900159124Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.903188142Z level=info msg="Executing migration" id="create user table v2" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.903679609Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.904106411Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.904579166Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-02-10T05:44:22.983 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.904976194Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.905677925Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.908400546Z level=info msg="Executing migration" id="Update user table charset" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.908699897Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.911614664Z level=info msg="Executing migration" id="Add missing user data" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.91198481Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.912424591Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.912841445Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.913276508Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.914457424Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.914878057Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.915354737Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.915803668Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-02-10T05:44:22.984 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.916664961Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.917585924Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.918300997Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.919139962Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.920034435Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.920865871Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.921335836Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.922099106Z level=info msg="Executing migration" id="create temp_user v2" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.922538291Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.922923784Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.923380258Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.923788161Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.924219928Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.924570055Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-02-10T05:44:22.985 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.925396908Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.925759241Z level=info msg="Executing migration" id="create star table" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.926209583Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.927189997Z level=info msg="Executing migration" id="create org table v1" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.92792196Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.9287551Z level=info msg="Executing migration" id="create org_user table v1" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.929626146Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.930634537Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-02-10T05:44:22.986 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.931365773Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.932243969Z level=info msg="Executing migration" id="Update org table charset" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.933141802Z level=info msg="Executing migration" id="Update org_user table charset" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.933929366Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.934919589Z level=info msg="Executing migration" id="create dashboard table" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.93566647Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.936537806Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.937388005Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.938240718Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.939109012Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.940058329Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.940980784Z level=info msg="Executing migration" id="create dashboard v2" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.943779699Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.946693804Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-02-10T05:44:22.987 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.949508025Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.952393674Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.955151966Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.957875024Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.960680355Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.963652212Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.966288532Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.969069549Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.971951556Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.9747612Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.977567494Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-02-10T05:44:22.988 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.980274527Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-02-10T05:44:23.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:22 smithi029 ceph-mon[25692]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:23.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:22 smithi029 ceph-mon[25692]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:23.175 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:22 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:22 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:22 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:22 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-10T05:44:23.230 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.983127381Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-02-10T05:44:23.230 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.98590257Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-02-10T05:44:23.230 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.988614149Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.991416387Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.994147719Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.996860873Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:22 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:22.999762388Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.002323553Z level=info msg="Executing migration" id="Update dashboard title length" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.005020745Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.007850822Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.010648479Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.013281357Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.0159241Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.018701155Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-02-10T05:44:23.231 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.021396222Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-02-10T05:44:23.232 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.024191622Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-02-10T05:44:23.232 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.026822274Z level=info msg="Executing migration" id="Add check_sum column" 2024-02-10T05:44:23.232 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.029515128Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-02-10T05:44:23.232 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.032414051Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.034996832Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.037729235Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.038463814Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.039507841Z level=info msg="Executing migration" id="create data_source table" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.040225809Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.04117576Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.041947158Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.042864438Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.043665214Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.044850277Z level=info msg="Executing migration" id="create data_source table v2" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.04577725Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-02-10T05:44:23.233 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.046525251Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.047385089Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.047756136Z level=info msg="Executing migration" id="Add column with_credentials" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.04856373Z level=info msg="Executing migration" id="Add secure json data column" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.049549966Z level=info msg="Executing migration" id="Update data_source table charset" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.050307591Z level=info msg="Executing migration" id="Update initial version to 1" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.050633472Z level=info msg="Executing migration" id="Add read_only data column" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.051515985Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.052334106Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.053263929Z level=info msg="Executing migration" id="Add uid column" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.05390828Z level=info msg="Executing migration" id="Update uid value" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.054894978Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.055580951Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-02-10T05:44:23.234 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.056607683Z level=info msg="Executing migration" id="create api_key table" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.057122151Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.058137517Z level=info msg="Executing migration" id="add index api_key.key" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.058843887Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.059846049Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.060428766Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.061275138Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.061720125Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.062722399Z level=info msg="Executing migration" id="create api_key table v2" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.063653631Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.064570497Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.065287395Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.066280833Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-02-10T05:44:23.235 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.068975929Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.071814297Z level=info msg="Executing migration" id="Update api_key table charset" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.074764779Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.077302665Z level=info msg="Executing migration" id="Add service account foreign key" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.080148022Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.082966205Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.085839475Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.088718656Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.091481129Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.094595549Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.097378867Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.100126293Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.102857103Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-02-10T05:44:23.236 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.105770503Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.108549081Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.111212894Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.113970264Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.116692826Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.119603492Z level=info msg="Executing migration" id="create quota table v1" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.12221689Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.125053344Z level=info msg="Executing migration" id="Update quota table charset" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.127691139Z level=info msg="Executing migration" id="create plugin_setting table" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.130370856Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.133176362Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.135834351Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.136167685Z level=info msg="Executing migration" id="create session table" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.136655198Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-02-10T05:44:23.237 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.136978177Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.137351084Z level=info msg="Executing migration" id="create playlist table v2" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.137758106Z level=info msg="Executing migration" id="create playlist item table v2" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.13817371Z level=info msg="Executing migration" id="Update playlist table charset" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.138477057Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.138756969Z level=info msg="Executing migration" id="drop preferences table v2" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.139078172Z level=info msg="Executing migration" id="drop preferences table v3" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.139401648Z level=info msg="Executing migration" id="create preferences table v3" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.139799658Z level=info msg="Executing migration" id="Update preferences table charset" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.140115207Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.140736295Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.141068647Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-02-10T05:44:23.238 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.141701192Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-02-10T05:44:23.239 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.142315486Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-02-10T05:44:23.239 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.14263423Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-02-10T05:44:23.239 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.143072139Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-02-10T05:44:23.239 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.143549979Z level=info msg="Executing migration" id="create alert table v1" 2024-02-10T05:44:23.239 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.144344685Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.145097159Z level=info msg="Executing migration" id="add index alert state" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.145525283Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.145943724Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.146331825Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.146754561Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.14717747Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.148621192Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.14902354Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.149828563Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.150186923Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.15055781Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.150928978Z level=info msg="Executing migration" id="Add column is_default" 2024-02-10T05:44:23.240 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.151627248Z level=info msg="Executing migration" id="Add column frequency" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.152307057Z level=info msg="Executing migration" id="Add column send_reminder" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.152965193Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.153600259Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.153999845Z level=info msg="Executing migration" id="Update alert table charset" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.154281129Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.154538747Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.154898666Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.155307562Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.155655701Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.156085433Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.156499384Z level=info msg="Executing migration" id="Add for to alert table" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.157326669Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.158100429Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-02-10T05:44:23.241 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.159062849Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.15971742Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.160669942Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.161534145Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.162375895Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.163225204Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.164152916Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.164884112Z level=info msg="Executing migration" id="create annotation table v5" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.165738382Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.16655927Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.167348381Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.168251499Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.16895614Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-02-10T05:44:23.242 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.169779244Z level=info msg="Executing migration" id="Update annotation table charset" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.170646047Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.171444884Z level=info msg="Executing migration" id="Drop category_id index" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.172335497Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.173097756Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.17396944Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.176770382Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.179181301Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.181993148Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.184735772Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.187440669Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.190099629Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.192954999Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.195631359Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-02-10T05:44:23.243 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.19817959Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.201104069Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.203712018Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.206538364Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.209205554Z level=info msg="Executing migration" id="Add epoch_end column" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.211830834Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.214417621Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.217300222Z level=info msg="Executing migration" id="Move region to single row" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.219990384Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.222517267Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.225137948Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.227751783Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.230470506Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-02-10T05:44:23.244 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.233090754Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-02-10T05:44:23.245 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.235751848Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-02-10T05:44:23.247 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:22 smithi110 ceph-mon[30869]: pgmap v22: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:23.247 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:22 smithi110 ceph-mon[30869]: from='client.14541 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:23.247 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:22 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.247 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:22 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.247 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:22 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.247 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:22 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:23.482 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.238400142Z level=info msg="Executing migration" id="create test_data table" 2024-02-10T05:44:23.482 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.24102167Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.241486183Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.24409321Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.246838606Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.249495963Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.252104054Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.254657199Z level=info msg="Executing migration" id="create team table" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.257248604Z level=info msg="Executing migration" id="add index team.org_id" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.259915261Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.262682227Z level=info msg="Executing migration" id="create team member table" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.265107804Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.26775135Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.268431765Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.269215256Z level=info msg="Executing migration" id="Add column email to team table" 2024-02-10T05:44:23.483 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.270041651Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.27082568Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.271620673Z level=info msg="Executing migration" id="create dashboard acl table" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.272426209Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.273242331Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.273655536Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.274067037Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.274922202Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.275337724Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.275733317Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.276145103Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.276533905Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.276797535Z level=info msg="Executing migration" id="create tag table" 2024-02-10T05:44:23.484 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.277182648Z level=info msg="Executing migration" id="add index tag.key_value" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.27758499Z level=info msg="Executing migration" id="create login attempt table" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.277956604Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.278370978Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.278733548Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.280563588Z level=info msg="Executing migration" id="create login_attempt v2" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.280944529Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.281381321Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.281660982Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-02-10T05:44:23.485 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.28199309Z level=info msg="Executing migration" id="create user auth table" 2024-02-10T05:44:23.486 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.282394864Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.282796976Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.283092965Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.283802264Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.284591023Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.285347053Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.286078734Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.286513582Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.287250997Z level=info msg="Executing migration" id="create server_lock table" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.2876271Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.28888084Z level=info msg="Executing migration" id="create user auth token table" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.291662911Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.294388138Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-02-10T05:44:23.487 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.297045484Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.297503331Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.300336672Z level=info msg="Executing migration" id="create cache_data table" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.30311158Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.305733864Z level=info msg="Executing migration" id="create short_url table v1" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.306188586Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.30910393Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.311763564Z level=info msg="Executing migration" id="delete alert_definition table" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.314318958Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.31476892Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.317528302Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.318265501Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.319255247Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-02-10T05:44:23.488 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.31977708Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.320570861Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.321417867Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.322246929Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.323050469Z level=info msg="Executing migration" id="drop alert_definition table" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.323887578Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.324794163Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.32554545Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.325971057Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.326391194Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.326788657Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.327220739Z level=info msg="Executing migration" id="create alert_instance table" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.327675212Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.328506744Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-02-10T05:44:23.489 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.328947514Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.329747181Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.330590614Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.331545766Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.334948963Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.338238353Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.339223058Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.339963533Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.340770228Z level=info msg="Executing migration" id="create alert_rule table" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.341600391Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.342553929Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.34332245Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.343744811Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.344647286Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-02-10T05:44:23.490 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.345665413Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.346496968Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.347369311Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.347779653Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.348730393Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.351429973Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.354365686Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.354819512Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.357602285Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.360439009Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.363138723Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.363583309Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.366526782Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.369156302Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-02-10T05:44:23.491 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.371907259Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.372789282Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.375643751Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.378476131Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.381195474Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.382034428Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.384895399Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.38746455Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.39025901Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.390572629Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.39333149Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.396205444Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.398595768Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-02-10T05:44:23.492 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.399014728Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.401727662Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.404641386Z level=info msg="Executing migration" id="create provenance_type table" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.40708669Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.407521851Z level=info msg="Executing migration" id="create alert_image table" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.410353397Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.413242865Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.415914876Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.416336187Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.419073758Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.419246271Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.419516607Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.421839216Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.422381812Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-02-10T05:44:23.493 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.422491542Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.422603861Z level=info msg="alerts found to migrate" alerts=0 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.424343562Z level=info msg="Executing migration" id="create library_element table v1" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.42479341Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.425706944Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.426374287Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.427188843Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.428138335Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.428509048Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.429336471Z level=info msg="Executing migration" id="create data_keys table" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.429756616Z level=info msg="Executing migration" id="create secrets table" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.430183403Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.434444198Z level=info msg="Executing migration" id="add name column into data_keys" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.435377489Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-02-10T05:44:23.494 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.436295538Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.440591633Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.444774341Z level=info msg="Executing migration" id="create kv_store table v1" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.445187788Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.44613949Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.446851593Z level=info msg="Executing migration" id="create permission table" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.44773811Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.448234805Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.449085619Z level=info msg="Executing migration" id="create role table" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.45005471Z level=info msg="Executing migration" id="add column display_name" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.45107964Z level=info msg="Executing migration" id="add column group_name" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.452019765Z level=info msg="Executing migration" id="add index role.org_id" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.452905101Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.453822414Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-02-10T05:44:23.495 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.454700323Z level=info msg="Executing migration" id="create team role table" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.455264164Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.456120309Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.456969864Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.457899138Z level=info msg="Executing migration" id="create user role table" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.458328031Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.459285273Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.460186645Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.461021744Z level=info msg="Executing migration" id="create builtin role table" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.461521621Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.464437843Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.467172348Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.470017131Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-02-10T05:44:23.496 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.470582999Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-02-10T05:44:23.497 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.473447294Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-02-10T05:44:23.497 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.476187722Z level=info msg="Executing migration" id="add unique index role.uid" 2024-02-10T05:44:23.497 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.478913965Z level=info msg="Executing migration" id="create seed assignment table" 2024-02-10T05:44:23.497 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.479347194Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-02-10T05:44:23.497 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.482458234Z level=info msg="Executing migration" id="add column hidden to role table" 2024-02-10T05:44:23.497 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.485130014Z level=info msg="Executing migration" id="create query_history table v1" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.487820862Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.488283468Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.491047225Z level=info msg="Executing migration" id="teams permissions migration" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.494087036Z level=info msg="Executing migration" id="dashboard permissions" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.496750834Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.497145151Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.49745732Z level=info msg="Executing migration" id="alerting notification permissions" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.497801178Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-02-10T05:44:23.910 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.498200169Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.498639462Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.499603023Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.499919046Z level=info msg="Executing migration" id="create correlation table v1" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.50033536Z level=info msg="Executing migration" id="add index correlations.uid" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.500771498Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.501221662Z level=info msg="Executing migration" id="add correlation config column" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.502223945Z level=info msg="Executing migration" id="create entity_events table" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.502699436Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.503148274Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.503244355Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.503534211Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.503626959Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:44:23.911 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.503923744Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.504364712Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.504769843Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.505292272Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.505754629Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.506573618Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.507474895Z level=info msg="Executing migration" id="Drop public config table" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.508175879Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.510818332Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.51371319Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.516607036Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.519153594Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.522137867Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-02-10T05:44:23.912 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.525009098Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-02-10T05:44:23.913 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.527525267Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-02-10T05:44:23.913 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.53037086Z level=info msg="Executing migration" id="add share column" 2024-02-10T05:44:23.913 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.533088961Z level=info msg="Executing migration" id="create default alerting folders" 2024-02-10T05:44:23.913 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.535979915Z level=info msg="Executing migration" id="create file table" 2024-02-10T05:44:23.913 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.538622516Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-02-10T05:44:23.913 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.541184449Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-02-10T05:44:23.913 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.541646163Z level=info msg="Executing migration" id="create file_meta table" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.543941329Z level=info msg="Executing migration" id="file table idx: path key" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.546724027Z level=info msg="Executing migration" id="set path collation in file table" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.549554071Z level=info msg="Executing migration" id="managed permissions migration" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.552251762Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.554977343Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.557665654Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.560108687Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.562765596Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.56520552Z level=info msg="Executing migration" id="update group index for alert rules" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.567836543Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.570282883Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-02-10T05:44:23.914 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.57300342Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.575554445Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.578034266Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.580659108Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.590508255Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.592967496Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.595629351Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.598952377Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.601634389Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.604070608Z level=info msg="Executing migration" id="create folder table" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.606710011Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.609090905Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.609951073Z level=info msg="Executing migration" id="Update folder title length" 2024-02-10T05:44:23.915 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.610573874Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=migrator t=2024-02-10T05:44:23.611419124Z level=info msg="migrations completed" performed=485 skipped=0 duration=718.502087ms 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=sqlstore t=2024-02-10T05:44:23.611804162Z level=info msg="Created default organization" 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=plugin.loader t=2024-02-10T05:44:23.665994071Z level=info msg="Plugin registered" pluginID=input 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=plugin.loader t=2024-02-10T05:44:23.692372694Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=plugin.loader t=2024-02-10T05:44:23.69240751Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=secrets t=2024-02-10T05:44:23.692440793Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=query_data t=2024-02-10T05:44:23.696151437Z level=info msg="Query Service initialization" 2024-02-10T05:44:23.916 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=live.push_http t=2024-02-10T05:44:23.698785249Z level=info msg="Live Push Gateway initialization" 2024-02-10T05:44:24.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:23 smithi029 ceph-mon[25692]: Deploying daemon node-exporter.a on smithi029 2024-02-10T05:44:24.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:23 smithi110 ceph-mon[30869]: Deploying daemon node-exporter.a on smithi029 2024-02-10T05:44:24.299 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:23 smithi105 ceph-mon[29854]: Deploying daemon node-exporter.a on smithi029 2024-02-10T05:44:24.659 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=infra.usagestats.collector t=2024-02-10T05:44:24.299714217Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-02-10T05:44:24.659 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=provisioning.datasources t=2024-02-10T05:44:24.300716762Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=provisioning.datasources t=2024-02-10T05:44:24.311675443Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=provisioning.alerting t=2024-02-10T05:44:24.322236969Z level=info msg="starting to provision alerting" 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=provisioning.alerting t=2024-02-10T05:44:24.322255262Z level=info msg="finished to provision alerting" 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=grafanaStorageLogger t=2024-02-10T05:44:24.322322378Z level=info msg="storage starting" 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=ngalert.state.manager t=2024-02-10T05:44:24.322442437Z level=info msg="Warming state cache for startup" 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=http.server t=2024-02-10T05:44:24.324894728Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=ngalert.state.manager t=2024-02-10T05:44:24.32723349Z level=info msg="State cache has been initialized" states=0 duration=4.78987ms 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=ngalert.multiorg.alertmanager t=2024-02-10T05:44:24.327271864Z level=info msg="starting MultiOrg Alertmanager" 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=ticker t=2024-02-10T05:44:24.327310382Z level=info msg=starting first_tick=2024-02-10T05:44:30Z 2024-02-10T05:44:24.660 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=sqlstore.transactions t=2024-02-10T05:44:24.33302997Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T05:44:24.661 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=sqlstore.transactions t=2024-02-10T05:44:24.335159907Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-02-10T05:44:24.661 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a[42971]: logger=sqlstore.transactions t=2024-02-10T05:44:24.345969884Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-02-10T05:44:25.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:24 smithi029 ceph-mon[25692]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:25.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:24 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:25.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:24 smithi110 ceph-mon[30869]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:25.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:24 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:25.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-mon[29854]: pgmap v23: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:25.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:24 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:25.684 INFO:teuthology.orchestra.run.smithi029.stdout:HOST ADDR LABELS STATUS 2024-02-10T05:44:25.684 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 172.21.15.29 2024-02-10T05:44:25.684 INFO:teuthology.orchestra.run.smithi029.stdout:smithi105 172.21.15.105 2024-02-10T05:44:25.685 INFO:teuthology.orchestra.run.smithi029.stdout:smithi110 172.21.15.110 2024-02-10T05:44:25.685 INFO:teuthology.orchestra.run.smithi029.stdout:3 hosts in cluster 2024-02-10T05:44:27.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-10T05:44:27.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:26 smithi029 ceph-mon[25692]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:27.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:26 smithi029 ceph-mon[25692]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:27.135 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:27 smithi029 bash[55790]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T05:44:27.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:26 smithi110 ceph-mon[30869]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:27.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:26 smithi110 ceph-mon[30869]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:27.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:26 smithi105 ceph-mon[29854]: from='client.14547 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:27.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:26 smithi105 ceph-mon[29854]: pgmap v24: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:27.557 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:27.599 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:27 smithi029 bash[55790]: Writing manifest to image destination 2024-02-10T05:44:27.884 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:27 smithi029 podman[55790]: 2024-02-10 05:44:27.583176593 +0000 UTC m=+2.710273565 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T05:44:27.884 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:27 smithi029 podman[55790]: 2024-02-10 05:44:27.709621014 +0000 UTC m=+2.836717985 container create 34f4b425c351bf5b4965b552519873649a62335ec378318169522ef2e76b54b8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:44:28.243 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 podman[55790]: 2024-02-10 05:44:28.141434434 +0000 UTC m=+3.268531443 container init 34f4b425c351bf5b4965b552519873649a62335ec378318169522ef2e76b54b8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:44:28.243 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 podman[55790]: 2024-02-10 05:44:28.146887861 +0000 UTC m=+3.273984835 container start 34f4b425c351bf5b4965b552519873649a62335ec378318169522ef2e76b54b8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:44:28.243 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.150Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T05:44:28.243 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.150Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T05:44:28.244 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T05:44:28.245 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T05:44:28.246 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T05:44:28.247 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.152Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.153Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a[56004]: ts=2024-02-10T05:44:28.153Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T05:44:28.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 bash[55790]: 34f4b425c351bf5b4965b552519873649a62335ec378318169522ef2e76b54b8 2024-02-10T05:44:28.495 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:44:28 smithi029 systemd[1]: Started Ceph node-exporter.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:44:29.520 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:29 smithi029 ceph-mon[25692]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:29.520 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:29 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.520 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:29 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.520 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:29 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.520 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:29 smithi029 ceph-mon[25692]: Deploying daemon node-exporter.b on smithi105 2024-02-10T05:44:29.520 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:29 smithi029 ceph-mon[25692]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:29.659 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:29 smithi105 ceph-mon[29854]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:29.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:29 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:29 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:29 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:29 smithi105 ceph-mon[29854]: Deploying daemon node-exporter.b on smithi105 2024-02-10T05:44:29.660 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:29 smithi105 ceph-mon[29854]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:29.739 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:29 smithi110 ceph-mon[30869]: pgmap v25: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:29.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:29 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:29 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:29 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' 2024-02-10T05:44:29.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:29 smithi110 ceph-mon[30869]: Deploying daemon node-exporter.b on smithi105 2024-02-10T05:44:29.740 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:29 smithi110 ceph-mon[30869]: from='mgr.14424 172.21.15.29:0/1974057045' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-02-10T05:44:30.006 INFO:teuthology.orchestra.run.smithi029.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-10T05:44:30.006 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7512009B480DGN 447G No 2m ago Has a FileSystem, LVM detected 2024-02-10T05:44:30.006 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme1n1 ssd Linux_16be90b6ca8db55c4ed5 89.4G Yes 2m ago 2024-02-10T05:44:30.006 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme2n1 ssd Linux_9486851be75fab6f415f 89.4G Yes 2m ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme3n1 ssd Linux_0a99aa7d45ac615615d4 89.4G Yes 2m ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi029 /dev/nvme4n1 ssd Linux_e3154ab83580eaccb4d7 89.4G No 2m ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 110s ago Has a FileSystem, LVM detected 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi105 /dev/nvme1n1 ssd Linux_cff449fa837008a84fc8 89.4G Yes 110s ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi105 /dev/nvme2n1 ssd Linux_d2829cba1868a95de4ca 89.4G Yes 110s ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi105 /dev/nvme3n1 ssd Linux_dbb18f21dd8b19ca220e 89.4G Yes 110s ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi105 /dev/nvme4n1 ssd Linux_34583723faaf7f26cdcb 89.4G No 110s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi110 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7476002T480DGN 447G No 86s ago Has a FileSystem, LVM detected 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi110 /dev/nvme1n1 ssd Linux_761888bf8420e14a1f25 89.4G Yes 86s ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi110 /dev/nvme2n1 ssd Linux_e1aea55af5723b83a713 89.4G Yes 86s ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi110 /dev/nvme3n1 ssd Linux_ac739ecbfb90b66518ec 89.4G Yes 86s ago 2024-02-10T05:44:30.007 INFO:teuthology.orchestra.run.smithi029.stdout:smithi110 /dev/nvme4n1 ssd Linux_633089ac8c83c16ab9fb 89.4G No 86s ago Has a FileSystem, Insufficient space (<10 extents) on vgs, LVM detected 2024-02-10T05:44:30.159 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:29 smithi105 bash[43240]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-02-10T05:44:30.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls --format yaml' 2024-02-10T05:44:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:30.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:30 smithi029 ceph-mon[25692]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:30.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:30 smithi029 ceph-mon[25692]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:31.110 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:30 smithi105 ceph-mon[29854]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:31.110 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:30 smithi105 ceph-mon[29854]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:31.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:30 smithi110 ceph-mon[30869]: pgmap v26: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:31.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:30 smithi110 ceph-mon[30869]: from='client.14553 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:31.409 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:31 smithi105 bash[43240]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-02-10T05:44:31.909 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:31 smithi105 bash[43240]: Writing manifest to image destination 2024-02-10T05:44:32.313 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:32 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: ::ffff:172.21.15.105 - - [10/Feb/2024:05:44:32] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: alertmanager 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: alertmanager 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout:placement: 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout: count: 1 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout: hosts: 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi029=a 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout: created: '2024-02-10T05:43:18.738186Z' 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout: ports: 2024-02-10T05:44:32.792 INFO:teuthology.orchestra.run.smithi029.stdout: - 9093 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: - 9094 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: running: 0 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: size: 1 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout:events: 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout:- 2024-02-10T05:44:09.102160Z service:alertmanager [INFO] "service was created" 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: grafana 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: grafana 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout:placement: 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: count: 1 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: hosts: 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi105=a 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout:spec: 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: anonymous_access: true 2024-02-10T05:44:32.793 INFO:teuthology.orchestra.run.smithi029.stdout: protocol: https 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: created: '2024-02-10T05:43:21.384438Z' 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: ports: 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: - 3000 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: running: 0 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: size: 1 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout:events: 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout:- 2024-02-10T05:44:22.902765Z service:grafana [INFO] "service was created" 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: mgr 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: mgr 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout:placement: 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: count: 3 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: hosts: 2024-02-10T05:44:32.794 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi029=y 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi105=x 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi110=z 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: created: '2024-02-10T05:41:44.538613Z' 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: last_refresh: '2024-02-10T05:43:45.507931Z' 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: running: 3 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: size: 3 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: mon 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: mon 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout:placement: 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: count: 3 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: hosts: 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi029:172.21.15.29=a 2024-02-10T05:44:32.795 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi105:172.21.15.105=b 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi110:172.21.15.110=c 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: created: '2024-02-10T05:41:15.615746Z' 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: last_refresh: '2024-02-10T05:43:45.507808Z' 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: running: 3 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: size: 3 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: node-exporter 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: node-exporter 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout:placement: 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: count: 3 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: hosts: 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi029=a 2024-02-10T05:44:32.796 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi105=b 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi110=c 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout: created: '2024-02-10T05:43:15.948091Z' 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout: ports: 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout: - 9100 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout: running: 0 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout: size: 3 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout:events: 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout:- 2024-02-10T05:44:28.332042Z service:node-exporter [INFO] "service was created" 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: osd 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: osd 2024-02-10T05:44:32.797 INFO:teuthology.orchestra.run.smithi029.stdout:unmanaged: true 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout:spec: 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: filter_logic: AND 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: objectstore: bluestore 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: container_image_id: c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: container_image_name: quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: last_refresh: '2024-02-10T05:43:45.508027Z' 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: running: 3 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: size: 3 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: prometheus 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: prometheus 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout:placement: 2024-02-10T05:44:32.798 INFO:teuthology.orchestra.run.smithi029.stdout: count: 1 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: hosts: 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi105=a 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: created: '2024-02-10T05:43:13.361419Z' 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: last_refresh: '2024-02-10T05:43:46.295876Z' 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: ports: 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: - 9095 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: running: 1 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: size: 1 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout:--- 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout:service_type: rgw 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout:service_id: foo.a 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout:service_name: rgw.foo.a 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout:placement: 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: count: 1 2024-02-10T05:44:32.799 INFO:teuthology.orchestra.run.smithi029.stdout: hosts: 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout: - smithi029=foo.a 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout:status: 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout: created: '2024-02-10T05:43:14.492760Z' 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout: last_refresh: '2024-02-10T05:43:53.801270Z' 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout: ports: 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout: - 80 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout: running: 1 2024-02-10T05:44:32.800 INFO:teuthology.orchestra.run.smithi029.stdout: size: 1 2024-02-10T05:44:33.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:32 smithi029 ceph-mon[25692]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:33.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:32 smithi105 ceph-mon[29854]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:33.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:32 smithi110 ceph-mon[30869]: pgmap v27: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:33.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 65498394-c7d6-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-02-10T05:44:33.698 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/mon.a/config 2024-02-10T05:44:34.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:33 smithi029 ceph-mon[25692]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-10T05:44:34.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:33 smithi105 ceph-mon[29854]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-10T05:44:34.159 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:33 smithi105 podman[43240]: 2024-02-10 05:44:33.728915013 +0000 UTC m=+4.346072007 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-02-10T05:44:34.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:33 smithi110 ceph-mon[30869]: from='client.14559 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "yaml"}]: dispatch 2024-02-10T05:44:34.909 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:34 smithi105 podman[43240]: 2024-02-10 05:44:34.635567399 +0000 UTC m=+5.252724383 container create f0cb4a04b6f0691c02a84dd5b74dbd276cb190a0c2794a4e459eb677eea9190e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:44:35.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:34 smithi029 ceph-mon[25692]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:35.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:34 smithi110 ceph-mon[30869]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:35.409 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:34 smithi105 ceph-mon[29854]: pgmap v28: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:35.962 INFO:teuthology.orchestra.run.smithi029.stdout:osd 3 50s ago - 2024-02-10T05:44:36.699 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-10T05:44:36.709 INFO:tasks.cephadm:Teardown begin 2024-02-10T05:44:36.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:44:36.741 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:44:36.772 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 podman[43240]: 2024-02-10 05:44:36.417141776 +0000 UTC m=+7.034298759 container init f0cb4a04b6f0691c02a84dd5b74dbd276cb190a0c2794a4e459eb677eea9190e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:44:36.772 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 podman[43240]: 2024-02-10 05:44:36.423054123 +0000 UTC m=+7.040211107 container start f0cb4a04b6f0691c02a84dd5b74dbd276cb190a0c2794a4e459eb677eea9190e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:44:36.772 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.427Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-02-10T05:44:36.772 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.427Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-02-10T05:44:36.772 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.427Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-02-10T05:44:36.772 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.427Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-02-10T05:44:36.772 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=arp 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=bcache 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=bonding 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=btrfs 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=conntrack 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=cpu 2024-02-10T05:44:36.773 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=diskstats 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=dmi 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=edac 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=entropy 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=filefd 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=filesystem 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=hwmon 2024-02-10T05:44:36.774 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=infiniband 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=ipvs 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=loadavg 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=mdadm 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=meminfo 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=netclass 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=netdev 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=netstat 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=nfs 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=nfsd 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=nvme 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=os 2024-02-10T05:44:36.775 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=pressure 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=rapl 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=schedstat 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=selinux 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=sockstat 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=softnet 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=stat 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=tapestats 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=textfile 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=time 2024-02-10T05:44:36.776 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-02-10T05:44:36.777 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.428Z caller=node_exporter.go:117 level=info collector=uname 2024-02-10T05:44:36.777 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.429Z caller=node_exporter.go:117 level=info collector=vmstat 2024-02-10T05:44:36.777 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.429Z caller=node_exporter.go:117 level=info collector=xfs 2024-02-10T05:44:36.777 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.429Z caller=node_exporter.go:117 level=info collector=zfs 2024-02-10T05:44:36.777 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.429Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-02-10T05:44:36.777 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b[43291]: ts=2024-02-10T05:44:36.429Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-02-10T05:44:36.778 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:44:36.815 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-10T05:44:36.815 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:44:36.831 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:44:36.848 DEBUG:teuthology.orchestra.run.smithi110:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-10T05:44:36.871 INFO:tasks.cephadm:Stopping all daemons... 2024-02-10T05:44:36.871 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-10T05:44:36.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a 2024-02-10T05:44:36.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:36 smithi029 ceph-mon[25692]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:36.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:36 smithi029 ceph-mon[25692]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:37.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-mon[29854]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:37.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:36 smithi105 ceph-mon[29854]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:37.208 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:36 smithi029 systemd[1]: Stopping Ceph mon.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:37.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:36 smithi110 ceph-mon[30869]: pgmap v29: 129 pgs: 129 active+clean; 582 KiB data, 102 MiB used, 268 GiB / 268 GiB avail 2024-02-10T05:44:37.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:36 smithi110 ceph-mon[30869]: from='client.14565 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-10T05:44:37.575 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a[25669]: 2024-02-10T05:44:37.207+0000 7f7170294700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:44:37.575 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a[25669]: 2024-02-10T05:44:37.207+0000 7f7170294700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-10T05:44:37.576 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:37 smithi029 podman[56669]: 2024-02-10 05:44:37.279403819 +0000 UTC m=+0.211165778 container died 2013fc10e01e409a0041c30ae853b6ad24da5e6c2170216e7c86ad282d4d128f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-10T05:44:37.576 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:37] ENGINE Bus STOPPING 2024-02-10T05:44:37.576 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-02-10T05:44:37.576 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:37] ENGINE Bus STOPPED 2024-02-10T05:44:37.576 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:37] ENGINE Bus STARTING 2024-02-10T05:44:37.576 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:37] ENGINE Serving on http://:::9283 2024-02-10T05:44:37.576 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:37 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: [10/Feb/2024:05:44:37] ENGINE Bus STARTED 2024-02-10T05:44:37.659 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:37 smithi105 bash[43240]: f0cb4a04b6f0691c02a84dd5b74dbd276cb190a0c2794a4e459eb677eea9190e 2024-02-10T05:44:37.659 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:44:37 smithi105 systemd[1]: Started Ceph node-exporter.b for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:44:37.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:37 smithi029 podman[56669]: 2024-02-10 05:44:37.683409905 +0000 UTC m=+0.615171863 container cleanup 2013fc10e01e409a0041c30ae853b6ad24da5e6c2170216e7c86ad282d4d128f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-10T05:44:37.884 INFO:journalctl@ceph.mon.a.smithi029.stdout:Feb 10 05:44:37 smithi029 bash[56669]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-a 2024-02-10T05:44:38.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.a.service' 2024-02-10T05:44:38.160 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:44:38.160 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-10T05:44:38.160 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-02-10T05:44:38.161 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.b 2024-02-10T05:44:38.456 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:38 smithi105 systemd[1]: Stopping Ceph mon.b for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:38.887 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b[29831]: 2024-02-10T05:44:38.456+0000 7f3133685700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:44:38.888 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:38 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b[29831]: 2024-02-10T05:44:38.456+0000 7f3133685700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-10T05:44:38.888 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:38 smithi105 podman[43403]: 2024-02-10 05:44:38.630759156 +0000 UTC m=+0.274977640 container died 01e2d35d4f7afa83895302f4e47ff3ba2166453cb32f98e6e1b62b6c02ffcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-10T05:44:39.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:39 smithi105 podman[43403]: 2024-02-10 05:44:39.004010126 +0000 UTC m=+0.648228608 container cleanup 01e2d35d4f7afa83895302f4e47ff3ba2166453cb32f98e6e1b62b6c02ffcdba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:44:39.159 INFO:journalctl@ceph.mon.b.smithi105.stdout:Feb 10 05:44:39 smithi105 bash[43403]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-b 2024-02-10T05:44:39.431 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.b.service' 2024-02-10T05:44:39.477 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:44:39.477 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-02-10T05:44:39.478 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-10T05:44:39.478 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.c 2024-02-10T05:44:39.765 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:39 smithi110 systemd[1]: Stopping Ceph mon.c for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:40.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:39 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c[30846]: 2024-02-10T05:44:39.763+0000 7f0ba6a51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:44:40.239 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:39 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c[30846]: 2024-02-10T05:44:39.763+0000 7f0ba6a51700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-10T05:44:40.240 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:39 smithi110 podman[44081]: 2024-02-10 05:44:39.99931277 +0000 UTC m=+0.327715447 container died 57ca5780fbb9144f1bbe5befd8780069e3412970eedeeb678865c08c70a6ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-02-10T05:44:40.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:40 smithi110 podman[44081]: 2024-02-10 05:44:40.403143946 +0000 UTC m=+0.731546627 container cleanup 57ca5780fbb9144f1bbe5befd8780069e3412970eedeeb678865c08c70a6ee99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-10T05:44:40.587 INFO:journalctl@ceph.mon.c.smithi110.stdout:Feb 10 05:44:40 smithi110 bash[44081]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-mon-c 2024-02-10T05:44:40.716 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mon.c.service' 2024-02-10T05:44:40.758 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:44:40.758 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-10T05:44:40.758 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-10T05:44:40.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.y 2024-02-10T05:44:41.051 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:40 smithi029 systemd[1]: Stopping Ceph mgr.y for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:41.360 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:41 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:44:41.050+0000 7fd742f32700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:44:41.360 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:41 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y[25952]: 2024-02-10T05:44:41.050+0000 7fd742f32700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-02-10T05:44:41.360 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:41 smithi029 podman[56811]: 2024-02-10 05:44:41.082438482 +0000 UTC m=+0.128389377 container died 9698232a4e27b6cac4881ae53b54fee31bc193ea3c7ad30c5de87f21ddd30779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-02-10T05:44:41.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:41 smithi029 podman[56811]: 2024-02-10 05:44:41.560483183 +0000 UTC m=+0.606434077 container cleanup 9698232a4e27b6cac4881ae53b54fee31bc193ea3c7ad30c5de87f21ddd30779 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=) 2024-02-10T05:44:41.634 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Feb 10 05:44:41 smithi029 bash[56811]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-y 2024-02-10T05:44:41.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-10T05:44:42.019 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:44:42.019 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-10T05:44:42.019 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-10T05:44:42.019 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.x 2024-02-10T05:44:42.314 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:44:42 smithi105 systemd[1]: Stopping Ceph mgr.x for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:42.608 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:44:42 smithi105 podman[43535]: 2024-02-10 05:44:42.314561052 +0000 UTC m=+0.104278807 container died b53f4d6dc2ba87b50f31df51a01d65364fb406911da5758f4e7b027397b259ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-02-10T05:44:42.900 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:44:42 smithi105 podman[43535]: 2024-02-10 05:44:42.69126332 +0000 UTC m=+0.480981080 container cleanup b53f4d6dc2ba87b50f31df51a01d65364fb406911da5758f4e7b027397b259ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:44:42.900 INFO:journalctl@ceph.mgr.x.smithi105.stdout:Feb 10 05:44:42 smithi105 bash[43535]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-x 2024-02-10T05:44:43.017 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-10T05:44:43.053 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:44:43.054 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-10T05:44:43.054 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-02-10T05:44:43.054 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.z 2024-02-10T05:44:43.356 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:44:43 smithi110 systemd[1]: Stopping Ceph mgr.z for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:43.641 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:44:43 smithi110 podman[44212]: 2024-02-10 05:44:43.356281074 +0000 UTC m=+0.115181439 container died ff4c8b64a0b2330b6faf23d3f8797cd83e53fa0552d13ecc7a8636c1cd66e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z, GIT_CLEAN=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:44:43.941 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:44:43 smithi110 podman[44212]: 2024-02-10 05:44:43.715677943 +0000 UTC m=+0.474578358 container cleanup ff4c8b64a0b2330b6faf23d3f8797cd83e53fa0552d13ecc7a8636c1cd66e91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:44:43.941 INFO:journalctl@ceph.mgr.z.smithi110.stdout:Feb 10 05:44:43 smithi110 bash[44212]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-mgr-z 2024-02-10T05:44:44.043 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@mgr.z.service' 2024-02-10T05:44:44.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:44:44.079 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-02-10T05:44:44.079 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-10T05:44:44.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.0 2024-02-10T05:44:44.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:44 smithi029 systemd[1]: Stopping Ceph osd.0 for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:44.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0[40338]: 2024-02-10T05:44:44.393+0000 7f222ef83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:44:44.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0[40338]: 2024-02-10T05:44:44.393+0000 7f222ef83700 -1 osd.0 31 *** Got signal Terminated *** 2024-02-10T05:44:44.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:44 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0[40338]: 2024-02-10T05:44:44.393+0000 7f222ef83700 -1 osd.0 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:44:50.129 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:49 smithi029 podman[56953]: 2024-02-10 05:44:49.834367612 +0000 UTC m=+5.564482236 container died 7fead61744937d9791b93e247a26d7eb1289aa7c8d9e8eef2ab82411112cbf5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-10T05:44:50.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:50 smithi029 podman[56953]: 2024-02-10 05:44:50.311944391 +0000 UTC m=+6.042059054 container cleanup 7fead61744937d9791b93e247a26d7eb1289aa7c8d9e8eef2ab82411112cbf5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, RELEASE=main-290212a) 2024-02-10T05:44:50.384 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:50 smithi029 bash[56953]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0 2024-02-10T05:44:50.884 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:50 smithi029 podman[56977]: 2024-02-10 05:44:50.589281249 +0000 UTC m=+0.744525892 container remove 7fead61744937d9791b93e247a26d7eb1289aa7c8d9e8eef2ab82411112cbf5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a) 2024-02-10T05:44:51.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:50 smithi029 podman[57073]: 2024-02-10 05:44:50.836620931 +0000 UTC m=+0.018577934 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:44:51.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:50 smithi029 podman[57073]: 2024-02-10 05:44:50.942582459 +0000 UTC m=+0.124539488 container create 2f30ac379cc3b74c145f701c52fdd1fabb90d5a2fe1ce3feb56709dee78eee4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:44:51.184 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:51 smithi029 podman[57073]: 2024-02-10 05:44:51.180239279 +0000 UTC m=+0.362196281 container init 2f30ac379cc3b74c145f701c52fdd1fabb90d5a2fe1ce3feb56709dee78eee4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-02-10T05:44:51.483 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:51 smithi029 podman[57073]: 2024-02-10 05:44:51.186924813 +0000 UTC m=+0.368881806 container start 2f30ac379cc3b74c145f701c52fdd1fabb90d5a2fe1ce3feb56709dee78eee4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:44:51.484 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:51 smithi029 podman[57073]: 2024-02-10 05:44:51.280347377 +0000 UTC m=+0.462304377 container attach 2f30ac379cc3b74c145f701c52fdd1fabb90d5a2fe1ce3feb56709dee78eee4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0-deactivate, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-10T05:44:51.826 INFO:journalctl@ceph.osd.0.smithi029.stdout:Feb 10 05:44:51 smithi029 podman[57073]: 2024-02-10 05:44:51.484265763 +0000 UTC m=+0.666222751 container died 2f30ac379cc3b74c145f701c52fdd1fabb90d5a2fe1ce3feb56709dee78eee4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T05:44:52.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.0.service' 2024-02-10T05:44:52.229 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:44:52.230 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-10T05:44:52.230 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-10T05:44:52.230 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.1 2024-02-10T05:44:52.522 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:52 smithi105 systemd[1]: Stopping Ceph osd.1 for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:44:52.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1[34474]: 2024-02-10T05:44:52.520+0000 7f7f28b62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:44:52.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1[34474]: 2024-02-10T05:44:52.520+0000 7f7f28b62700 -1 osd.1 31 *** Got signal Terminated *** 2024-02-10T05:44:52.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:52 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1[34474]: 2024-02-10T05:44:52.520+0000 7f7f28b62700 -1 osd.1 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:44:58.278 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:58 smithi105 podman[43670]: 2024-02-10 05:44:58.016645427 +0000 UTC m=+5.594502475 container died de67ba3154ac48dd2dd2b9a3f1aa0364b77cb5fe320ebb08e3201037a2efa40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a) 2024-02-10T05:44:58.552 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:58 smithi105 podman[43670]: 2024-02-10 05:44:58.352111909 +0000 UTC m=+5.929968952 container cleanup de67ba3154ac48dd2dd2b9a3f1aa0364b77cb5fe320ebb08e3201037a2efa40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-02-10T05:44:58.553 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:58 smithi105 bash[43670]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1 2024-02-10T05:44:58.909 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:58 smithi105 podman[43701]: 2024-02-10 05:44:58.553112844 +0000 UTC m=+0.534207225 container remove de67ba3154ac48dd2dd2b9a3f1aa0364b77cb5fe320ebb08e3201037a2efa40c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:44:59.264 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:58 smithi105 podman[43798]: 2024-02-10 05:44:58.852807202 +0000 UTC m=+0.017368712 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:44:59.264 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:58 smithi105 podman[43798]: 2024-02-10 05:44:58.987151988 +0000 UTC m=+0.151713488 container create d9c57f07d57aa49303c33ef38e9bae33fdcce428ba784ea554fba451951a2255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=) 2024-02-10T05:44:59.264 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:59 smithi105 podman[43798]: 2024-02-10 05:44:59.181323004 +0000 UTC m=+0.345884518 container init d9c57f07d57aa49303c33ef38e9bae33fdcce428ba784ea554fba451951a2255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-10T05:44:59.264 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:59 smithi105 podman[43798]: 2024-02-10 05:44:59.187136606 +0000 UTC m=+0.351698111 container start d9c57f07d57aa49303c33ef38e9bae33fdcce428ba784ea554fba451951a2255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True) 2024-02-10T05:44:59.264 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:59 smithi105 podman[43798]: 2024-02-10 05:44:59.264479026 +0000 UTC m=+0.429040535 container attach d9c57f07d57aa49303c33ef38e9bae33fdcce428ba784ea554fba451951a2255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-10T05:44:59.659 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:44:59 smithi105 podman[43798]: 2024-02-10 05:44:59.477499912 +0000 UTC m=+0.642061406 container died d9c57f07d57aa49303c33ef38e9bae33fdcce428ba784ea554fba451951a2255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True) 2024-02-10T05:45:00.025 INFO:journalctl@ceph.osd.1.smithi105.stdout:Feb 10 05:45:00 smithi105 podman[43844]: 2024-02-10 05:45:00.009013036 +0000 UTC m=+0.526338979 container remove d9c57f07d57aa49303c33ef38e9bae33fdcce428ba784ea554fba451951a2255 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a) 2024-02-10T05:45:00.030 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.1.service' 2024-02-10T05:45:00.070 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:45:00.071 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-10T05:45:00.071 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-10T05:45:00.071 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.2 2024-02-10T05:45:00.371 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:00 smithi110 systemd[1]: Stopping Ceph osd.2 for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:45:00.739 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:00 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2[36372]: 2024-02-10T05:45:00.370+0000 7fcace317700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:45:00.739 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:00 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2[36372]: 2024-02-10T05:45:00.370+0000 7fcace317700 -1 osd.2 31 *** Got signal Terminated *** 2024-02-10T05:45:00.740 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:00 smithi110 ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2[36372]: 2024-02-10T05:45:00.370+0000 7fcace317700 -1 osd.2 31 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-10T05:45:06.035 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:05 smithi110 podman[44344]: 2024-02-10 05:45:05.76519525 +0000 UTC m=+5.504887810 container died 8b31da4147879d093ac81c3891ea33f0a13c0d15bbbd770064a2ee3e114cfd8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-10T05:45:06.385 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:06 smithi110 podman[44344]: 2024-02-10 05:45:06.134421815 +0000 UTC m=+5.874114379 container cleanup 8b31da4147879d093ac81c3891ea33f0a13c0d15bbbd770064a2ee3e114cfd8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-10T05:45:06.385 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:06 smithi110 bash[44344]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2 2024-02-10T05:45:06.739 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:06 smithi110 podman[44366]: 2024-02-10 05:45:06.38504862 +0000 UTC m=+0.620925853 container remove 8b31da4147879d093ac81c3891ea33f0a13c0d15bbbd770064a2ee3e114cfd8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.build-date=20240131) 2024-02-10T05:45:07.006 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:06 smithi110 podman[44459]: 2024-02-10 05:45:06.644619287 +0000 UTC m=+0.017668193 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-10T05:45:07.006 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:06 smithi110 podman[44459]: 2024-02-10 05:45:06.746426437 +0000 UTC m=+0.119475327 container create bffa2cad042f0ca1ae21dc1c86f78628fd0461cc674dc8d41af32f599f4b4956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-10T05:45:07.007 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:06 smithi110 podman[44459]: 2024-02-10 05:45:06.932011959 +0000 UTC m=+0.305060851 container init bffa2cad042f0ca1ae21dc1c86f78628fd0461cc674dc8d41af32f599f4b4956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-10T05:45:07.007 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:06 smithi110 podman[44459]: 2024-02-10 05:45:06.937167574 +0000 UTC m=+0.310216463 container start bffa2cad042f0ca1ae21dc1c86f78628fd0461cc674dc8d41af32f599f4b4956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240131, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True) 2024-02-10T05:45:07.007 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:07 smithi110 podman[44459]: 2024-02-10 05:45:07.006685793 +0000 UTC m=+0.379734679 container attach bffa2cad042f0ca1ae21dc1c86f78628fd0461cc674dc8d41af32f599f4b4956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-10T05:45:07.472 INFO:journalctl@ceph.osd.2.smithi110.stdout:Feb 10 05:45:07 smithi110 podman[44459]: 2024-02-10 05:45:07.227305446 +0000 UTC m=+0.600354335 container died bffa2cad042f0ca1ae21dc1c86f78628fd0461cc674dc8d41af32f599f4b4956 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.vendor=CentOS) 2024-02-10T05:45:07.784 DEBUG:teuthology.orchestra.run.smithi110:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@osd.2.service' 2024-02-10T05:45:07.824 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:45:07.824 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-10T05:45:07.824 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-02-10T05:45:07.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@rgw.foo.a 2024-02-10T05:45:08.131 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:07 smithi029 systemd[1]: Stopping Ceph rgw.foo.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:45:08.384 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a[47162]: 2024-02-10T05:45:08.131+0000 7fd7c1552700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-10T05:45:08.384 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:08 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a[47162]: 2024-02-10T05:45:08.131+0000 7fd7cfdc3ac0 -1 shutting down 2024-02-10T05:45:18.456 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:18 smithi029 bash[57205]: time="2024-02-10T05:45:18Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-02-10T05:45:18.457 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:18 smithi029 podman[57205]: 2024-02-10 05:45:18.157483052 +0000 UTC m=+10.135327349 container died 6b1c9238d08979e1a10aeda3af15434dee508cc6413ae947ea6bb1aa1c7dc1fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-02-10T05:45:18.794 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:18 smithi029 podman[57205]: 2024-02-10 05:45:18.655975056 +0000 UTC m=+10.633819352 container cleanup 6b1c9238d08979e1a10aeda3af15434dee508cc6413ae947ea6bb1aa1c7dc1fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-10T05:45:18.794 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:18 smithi029 bash[57205]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a 2024-02-10T05:45:19.049 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:18 smithi029 podman[57230]: 2024-02-10 05:45:18.891543475 +0000 UTC m=+0.729107910 container remove 6b1c9238d08979e1a10aeda3af15434dee508cc6413ae947ea6bb1aa1c7dc1fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-10T05:45:19.049 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Feb 10 05:45:18 smithi029 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-02-10T05:45:19.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@rgw.foo.a.service' 2024-02-10T05:45:19.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:45:19.091 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-02-10T05:45:19.091 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-02-10T05:45:19.092 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-65498394-c7d6-11ee-95b8-87774f69a715@prometheus.a 2024-02-10T05:45:19.409 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 systemd[1]: Stopping Ceph prometheus.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:45:19.752 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-02-10T05:45:19.752 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-02-10T05:45:19.752 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-02-10T05:45:19.752 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-02-10T05:45:19.752 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-02-10T05:45:19.752 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-02-10T05:45:19.753 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-02-10T05:45:19.753 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-02-10T05:45:19.753 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-02-10T05:45:19.753 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-02-10T05:45:19.753 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-02-10T05:45:19.753 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a[41087]: ts=2024-02-10T05:45:19.471Z caller=main.go:1147 level=info msg="See you next time!" 2024-02-10T05:45:19.753 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 podman[43927]: 2024-02-10 05:45:19.481535486 +0000 UTC m=+0.194921480 container died 7741dbf83128b587b21021c12990c19028cdaf6eb0a4d4c1171c4286f9562d02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:20.093 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 podman[43927]: 2024-02-10 05:45:19.851242619 +0000 UTC m=+0.564628622 container cleanup 7741dbf83128b587b21021c12990c19028cdaf6eb0a4d4c1171c4286f9562d02 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:20.094 INFO:journalctl@ceph.prometheus.a.smithi105.stdout:Feb 10 05:45:19 smithi105 bash[43927]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-prometheus-a 2024-02-10T05:45:20.214 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-65498394-c7d6-11ee-95b8-87774f69a715@prometheus.a.service' 2024-02-10T05:45:20.249 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-10T05:45:20.249 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-02-10T05:45:20.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 65498394-c7d6-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T05:45:20.439 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:45:21.384 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:21 smithi029 systemd[1]: Stopping Ceph alertmanager.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:45:21.857 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:21 smithi029 ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a[54387]: ts=2024-02-10T05:45:21.550Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-02-10T05:45:21.858 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:21 smithi029 podman[57562]: 2024-02-10 05:45:21.571479116 +0000 UTC m=+0.131690189 container died 9787d0a2b423d4cd723825af9b8c1581344e912b8e2ffe16b902095abe6c5454 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:22.134 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:22 smithi029 podman[57562]: 2024-02-10 05:45:22.058052266 +0000 UTC m=+0.618263339 container cleanup 9787d0a2b423d4cd723825af9b8c1581344e912b8e2ffe16b902095abe6c5454 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:22.134 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:22 smithi029 bash[57562]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a 2024-02-10T05:45:22.848 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:22 smithi029 podman[57579]: 2024-02-10 05:45:22.412984383 +0000 UTC m=+0.844650230 container remove 9787d0a2b423d4cd723825af9b8c1581344e912b8e2ffe16b902095abe6c5454 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:22.848 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:22 smithi029 podman[57579]: 2024-02-10 05:45:22.497254471 +0000 UTC m=+0.928920318 volume remove 38ab937cb7d3cefbac50d9ed0eaa9032e56bd6770132a4fca11310b4d93489ac 2024-02-10T05:45:22.848 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:22 smithi029 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-02-10T05:45:22.849 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Feb 10 05:45:22 smithi029 systemd[1]: Stopped Ceph alertmanager.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:45:23.134 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:22 smithi029 systemd[1]: Stopping Ceph node-exporter.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:45:23.134 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 podman[57697]: 2024-02-10 05:45:23.097169015 +0000 UTC m=+0.110185406 container died 34f4b425c351bf5b4965b552519873649a62335ec378318169522ef2e76b54b8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:23.634 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 podman[57697]: 2024-02-10 05:45:23.527526218 +0000 UTC m=+0.540542639 container cleanup 34f4b425c351bf5b4965b552519873649a62335ec378318169522ef2e76b54b8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:23.634 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 bash[57697]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a 2024-02-10T05:45:24.107 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 podman[57716]: 2024-02-10 05:45:23.748147599 +0000 UTC m=+0.656160690 container remove 34f4b425c351bf5b4965b552519873649a62335ec378318169522ef2e76b54b8 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-02-10T05:45:24.107 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T05:45:24.107 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-02-10T05:45:24.107 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 systemd[1]: Stopped Ceph node-exporter.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:45:24.107 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Feb 10 05:45:23 smithi029 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.a.service: Consumed 1.762s CPU time. 2024-02-10T05:45:24.599 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 65498394-c7d6-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T05:45:24.790 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:45:25.994 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:25 smithi105 systemd[1]: Stopping Ceph grafana.a for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:45:26.278 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:25 smithi105 podman[44276]: 2024-02-10 05:45:25.993923335 +0000 UTC m=+0.117683769 container died ddfd594a50f1c6466d2d05685562dd8d1f32524c019746e838b84873056b69ae (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, release=239.1651231664, description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, vcs-type=git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, version=8.5, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, name=ubi8) 2024-02-10T05:45:26.605 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:26 smithi105 podman[44276]: 2024-02-10 05:45:26.288559763 +0000 UTC m=+0.412320206 container cleanup ddfd594a50f1c6466d2d05685562dd8d1f32524c019746e838b84873056b69ae (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a, description=Ceph Grafana Container, release=239.1651231664, version=8.5, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, io.openshift.expose-services=, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git) 2024-02-10T05:45:26.605 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:26 smithi105 bash[44276]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a 2024-02-10T05:45:26.605 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:26 smithi105 podman[44295]: 2024-02-10 05:45:26.497866186 +0000 UTC m=+0.505373768 container remove ddfd594a50f1c6466d2d05685562dd8d1f32524c019746e838b84873056b69ae (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , vendor=Red Hat, Inc., version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vcs-type=git, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, release=239.1651231664, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container) 2024-02-10T05:45:26.606 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:26 smithi105 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-02-10T05:45:26.909 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:26 smithi105 systemd[1]: Stopping Ceph node-exporter.b for 65498394-c7d6-11ee-95b8-87774f69a715... 2024-02-10T05:45:26.909 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:26 smithi105 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-02-10T05:45:26.909 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:26 smithi105 systemd[1]: Stopped Ceph grafana.a for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:45:26.909 INFO:journalctl@ceph.grafana.a.smithi105.stdout:Feb 10 05:45:26 smithi105 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@grafana.a.service: Consumed 2.723s CPU time. 2024-02-10T05:45:27.358 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 podman[44412]: 2024-02-10 05:45:27.072326596 +0000 UTC m=+0.119684773 container died f0cb4a04b6f0691c02a84dd5b74dbd276cb190a0c2794a4e459eb677eea9190e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:45:27.358 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 podman[44412]: 2024-02-10 05:45:27.358827109 +0000 UTC m=+0.406185294 container cleanup f0cb4a04b6f0691c02a84dd5b74dbd276cb190a0c2794a4e459eb677eea9190e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:45:27.659 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 bash[44412]: ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b 2024-02-10T05:45:27.659 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 podman[44430]: 2024-02-10 05:45:27.543120437 +0000 UTC m=+0.475642280 container remove f0cb4a04b6f0691c02a84dd5b74dbd276cb190a0c2794a4e459eb677eea9190e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-65498394-c7d6-11ee-95b8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-02-10T05:45:27.659 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-02-10T05:45:28.159 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-02-10T05:45:28.159 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 systemd[1]: Stopped Ceph node-exporter.b for 65498394-c7d6-11ee-95b8-87774f69a715. 2024-02-10T05:45:28.159 INFO:journalctl@ceph.node-exporter.b.smithi105.stdout:Feb 10 05:45:27 smithi105 systemd[1]: ceph-65498394-c7d6-11ee-95b8-87774f69a715@node-exporter.b.service: Consumed 1.707s CPU time. 2024-02-10T05:45:28.394 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 65498394-c7d6-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-10T05:45:28.583 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:45:29.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:45:29.765 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:45:29.800 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-10T05:45:29.829 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-10T05:45:29.831 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi029/crash 2024-02-10T05:45:29.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash -- . 2024-02-10T05:45:29.873 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T05:45:29.873 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-02-10T05:45:29.875 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi105/crash 2024-02-10T05:45:29.876 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash -- . 2024-02-10T05:45:29.914 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T05:45:29.914 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2024-02-10T05:45:29.915 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi110/crash 2024-02-10T05:45:29.916 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash -- . 2024-02-10T05:45:29.951 INFO:teuthology.orchestra.run.smithi110.stderr:tar: /var/lib/ceph/65498394-c7d6-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-10T05:45:29.952 INFO:teuthology.orchestra.run.smithi110.stderr:tar: Error is not recoverable: exiting now 2024-02-10T05:45:29.953 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-10T05:45:29.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:45:29.983 INFO:teuthology.orchestra.run.smithi029.stdout:2024-02-10T05:41:35.387173+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:45:29.983 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-02-10T05:45:29.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[SEC\]' /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:45:30.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]' /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:45:30.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[WRN\]' /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-02-10T05:45:30.190 INFO:teuthology.orchestra.run.smithi029.stdout:2024-02-10T05:41:35.387173+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-02-10T05:45:30.190 INFO:tasks.cephadm:Compressing logs... 2024-02-10T05:45:30.191 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T05:45:30.232 DEBUG:teuthology.orchestra.run.smithi105:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T05:45:30.235 DEBUG:teuthology.orchestra.run.smithi110:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-10T05:45:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T05:45:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-02-10T05:45:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log/var/log/ceph/cephadm.log: 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.a.log: /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log: 57.8% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log 90.6% 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log: 82.1% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T05:45:30.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log: 85.0% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T05:45:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-10T05:45:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:: No such file or directory 2024-02-10T05:45:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T05:45:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.z.log 2024-02-10T05:45:30.263 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log: /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log: 90.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr: 80.5% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log: 89.5% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-10T05:45:30.264 INFO:teuthology.orchestra.run.smithi110.stderr: 84.6% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T05:45:30.265 INFO:teuthology.orchestra.run.smithi029.stderr: 90.6% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T05:45:30.266 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log: No such file or directory 2024-02-10T05:45:30.266 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-02-10T05:45:30.266 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-10T05:45:30.266 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-10T05:45:30.266 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-10T05:45:30.266 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-10T05:45:30.267 INFO:teuthology.orchestra.run.smithi110.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.c.log: 90.1% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.z.log.gz 2024-02-10T05:45:30.267 INFO:teuthology.orchestra.run.smithi110.stderr: 90.5% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T05:45:30.268 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log 2024-02-10T05:45:30.268 INFO:teuthology.orchestra.run.smithi105.stderr: 89.5% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-10T05:45:30.268 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-10T05:45:30.268 INFO:teuthology.orchestra.run.smithi105.stderr: 80.5% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-10T05:45:30.268 INFO:teuthology.orchestra.run.smithi105.stderr:gzip -5 --verbose -- /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-10T05:45:30.268 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log: /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log: 81.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-10T05:45:30.269 INFO:teuthology.orchestra.run.smithi105.stderr: 84.4% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-10T05:45:30.269 INFO:teuthology.orchestra.run.smithi105.stderr:/var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.b.log: 90.5% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-10T05:45:30.269 INFO:teuthology.orchestra.run.smithi105.stderr: 90.2% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-10T05:45:30.320 INFO:teuthology.orchestra.run.smithi110.stderr: 92.2% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-10T05:45:30.328 INFO:teuthology.orchestra.run.smithi105.stderr: 92.5% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-10T05:45:30.333 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-10T05:45:30.441 INFO:teuthology.orchestra.run.smithi029.stderr: 91.4% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-10T05:45:30.548 INFO:teuthology.orchestra.run.smithi110.stderr: 94.2% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-10T05:45:30.550 INFO:teuthology.orchestra.run.smithi110.stderr: 2024-02-10T05:45:30.550 INFO:teuthology.orchestra.run.smithi110.stderr:real 0m0.301s 2024-02-10T05:45:30.550 INFO:teuthology.orchestra.run.smithi110.stderr:user 0m0.355s 2024-02-10T05:45:30.550 INFO:teuthology.orchestra.run.smithi110.stderr:sys 0m0.028s 2024-02-10T05:45:30.562 INFO:teuthology.orchestra.run.smithi029.stderr: 94.2% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-10T05:45:30.564 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-02-10T05:45:30.564 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m0.321s 2024-02-10T05:45:30.565 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m0.565s 2024-02-10T05:45:30.565 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m0.041s 2024-02-10T05:45:30.591 INFO:teuthology.orchestra.run.smithi105.stderr: 94.3% -- replaced with /var/log/ceph/65498394-c7d6-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-10T05:45:30.593 INFO:teuthology.orchestra.run.smithi105.stderr: 2024-02-10T05:45:30.593 INFO:teuthology.orchestra.run.smithi105.stderr:real 0m0.344s 2024-02-10T05:45:30.593 INFO:teuthology.orchestra.run.smithi105.stderr:user 0m0.400s 2024-02-10T05:45:30.593 INFO:teuthology.orchestra.run.smithi105.stderr:sys 0m0.036s 2024-02-10T05:45:30.594 INFO:tasks.cephadm:Archiving logs... 2024-02-10T05:45:30.594 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi029/log 2024-02-10T05:45:30.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T05:45:30.733 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi105/log 2024-02-10T05:45:30.734 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T05:45:30.831 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi110/log 2024-02-10T05:45:30.832 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-10T05:45:30.921 INFO:tasks.cephadm:Removing cluster... 2024-02-10T05:45:30.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 65498394-c7d6-11ee-95b8-87774f69a715 --force 2024-02-10T05:45:31.105 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:45:31.219 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 65498394-c7d6-11ee-95b8-87774f69a715 --force 2024-02-10T05:45:31.400 INFO:teuthology.orchestra.run.smithi105.stdout:Deleting cluster with fsid: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:45:31.517 DEBUG:teuthology.orchestra.run.smithi110:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 65498394-c7d6-11ee-95b8-87774f69a715 --force 2024-02-10T05:45:31.696 INFO:teuthology.orchestra.run.smithi110.stdout:Deleting cluster with fsid: 65498394-c7d6-11ee-95b8-87774f69a715 2024-02-10T05:45:31.808 INFO:tasks.cephadm:Removing cephadm ... 2024-02-10T05:45:31.808 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T05:45:31.824 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T05:45:31.840 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-10T05:45:31.855 INFO:tasks.cephadm:Teardown complete 2024-02-10T05:45:31.856 DEBUG:teuthology.run_tasks:Unwinding manager nvme_loop 2024-02-10T05:45:31.866 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi029:/dev/vg_nvme/lv_1... 2024-02-10T05:45:31.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo nvme disconnect -n lv_1 2024-02-10T05:45:32.010 INFO:teuthology.orchestra.run.smithi029.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-10T05:45:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:32.012 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi029:/dev/vg_nvme/lv_2... 2024-02-10T05:45:32.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo nvme disconnect -n lv_2 2024-02-10T05:45:32.155 INFO:teuthology.orchestra.run.smithi029.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-10T05:45:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:32.157 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi029:/dev/vg_nvme/lv_3... 2024-02-10T05:45:32.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo nvme disconnect -n lv_3 2024-02-10T05:45:32.331 INFO:teuthology.orchestra.run.smithi029.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-10T05:45:32.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:32.333 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi029:/dev/vg_nvme/lv_4... 2024-02-10T05:45:32.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo nvme disconnect -n lv_4 2024-02-10T05:45:32.484 INFO:teuthology.orchestra.run.smithi029.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-10T05:45:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:32.486 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-02-10T05:45:32.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/scratch_devs 2024-02-10T05:45:32.513 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi105:/dev/vg_nvme/lv_1... 2024-02-10T05:45:32.513 DEBUG:teuthology.orchestra.run.smithi105:> sudo nvme disconnect -n lv_1 2024-02-10T05:45:32.657 INFO:teuthology.orchestra.run.smithi105.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-10T05:45:32.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:32.660 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi105:/dev/vg_nvme/lv_2... 2024-02-10T05:45:32.660 DEBUG:teuthology.orchestra.run.smithi105:> sudo nvme disconnect -n lv_2 2024-02-10T05:45:32.804 INFO:teuthology.orchestra.run.smithi105.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-10T05:45:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:32.806 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi105:/dev/vg_nvme/lv_3... 2024-02-10T05:45:32.806 DEBUG:teuthology.orchestra.run.smithi105:> sudo nvme disconnect -n lv_3 2024-02-10T05:45:32.963 INFO:teuthology.orchestra.run.smithi105.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-10T05:45:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:32.965 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi105:/dev/vg_nvme/lv_4... 2024-02-10T05:45:32.965 DEBUG:teuthology.orchestra.run.smithi105:> sudo nvme disconnect -n lv_4 2024-02-10T05:45:33.098 INFO:teuthology.orchestra.run.smithi105.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-10T05:45:33.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:33.100 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-02-10T05:45:33.100 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/scratch_devs 2024-02-10T05:45:33.131 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi110:/dev/vg_nvme/lv_1... 2024-02-10T05:45:33.131 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme disconnect -n lv_1 2024-02-10T05:45:33.284 INFO:teuthology.orchestra.run.smithi110.stdout:NQN:lv_1 disconnected 1 controller(s) 2024-02-10T05:45:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:33.288 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi110:/dev/vg_nvme/lv_2... 2024-02-10T05:45:33.288 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme disconnect -n lv_2 2024-02-10T05:45:33.427 INFO:teuthology.orchestra.run.smithi110.stdout:NQN:lv_2 disconnected 1 controller(s) 2024-02-10T05:45:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:33.429 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi110:/dev/vg_nvme/lv_3... 2024-02-10T05:45:33.429 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme disconnect -n lv_3 2024-02-10T05:45:33.612 INFO:teuthology.orchestra.run.smithi110.stdout:NQN:lv_3 disconnected 1 controller(s) 2024-02-10T05:45:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:33.614 INFO:tasks.nvme_loop:Disconnecting nvme_loop smithi110:/dev/vg_nvme/lv_4... 2024-02-10T05:45:33.614 DEBUG:teuthology.orchestra.run.smithi110:> sudo nvme disconnect -n lv_4 2024-02-10T05:45:33.793 INFO:teuthology.orchestra.run.smithi110.stdout:NQN:lv_4 disconnected 1 controller(s) 2024-02-10T05:45:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:33.795 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2024-02-10T05:45:33.795 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/scratch_devs 2024-02-10T05:45:33.821 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-10T05:45:33.831 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-10T05:45:33.831 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:45:33.834 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:45:33.836 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-10T05:45:33.846 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:45:33.849 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:45:33.850 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-02-10T05:45:33.850 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +609us[ +609us] +/- 91ms 2024-02-10T05:45:33.850 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -732us[ -725us] +/- 65ms 2024-02-10T05:45:33.850 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +440us[ +440us] +/- 92ms 2024-02-10T05:45:33.850 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:45:33.850 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:45:33.855 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:45:33.855 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-02-10T05:45:33.855 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +598us[ +596us] +/- 91ms 2024-02-10T05:45:33.855 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -722us[ -725us] +/- 65ms 2024-02-10T05:45:33.855 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +416us[ +413us] +/- 92ms 2024-02-10T05:45:33.856 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:45:33.875 INFO:teuthology.orchestra.run.smithi110.stderr:bash: line 1: ntpq: command not found 2024-02-10T05:45:33.879 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-10T05:45:33.879 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2024-02-10T05:45:33.879 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 3 +616us[ +619us] +/- 91ms 2024-02-10T05:45:33.879 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -710us[ -707us] +/- 65ms 2024-02-10T05:45:33.879 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +426us[ +429us] +/- 92ms 2024-02-10T05:45:33.879 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-02-10T05:45:33.880 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-10T05:45:33.889 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-10T05:45:33.890 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-10T05:45:33.898 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T05:45:33.972 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T05:45:34.045 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-10T05:45:34.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:45:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:34.144 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:45:34.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:34.176 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-10T05:45:34.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:34.205 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-10T05:45:34.214 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-10T05:45:34.268 INFO:teuthology.task.internal:Duration was 793.434940 seconds 2024-02-10T05:45:34.268 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-10T05:45:34.359 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-10T05:45:34.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T05:45:34.362 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T05:45:34.364 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-10T05:45:34.404 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:45:34.407 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:45:34.408 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-10T05:45:34.924 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-10T05:45:34.924 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-02-10T05:45:34.924 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T05:45:34.945 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-02-10T05:45:34.945 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T05:45:34.980 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2024-02-10T05:45:34.980 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-10T05:45:35.006 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-10T05:45:35.007 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T05:45:35.009 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T05:45:35.023 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-10T05:45:35.078 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-10T05:45:35.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:45:35.081 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:45:35.096 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-10T05:45:35.268 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-10T05:45:35.279 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-10T05:45:35.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T05:45:35.305 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T05:45:35.332 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-10T05:45:35.359 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-10T05:45:35.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:45:35.372 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:45:35.374 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:45:35.394 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-02-10T05:45:35.398 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-02-10T05:45:35.424 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2024-02-10T05:45:35.450 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:45:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:35.474 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:45:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:35.489 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-10T05:45:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-10T05:45:35.504 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-10T05:45:35.514 INFO:teuthology.task.internal:Transferring archived files... 2024-02-10T05:45:35.515 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi029 2024-02-10T05:45:35.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T05:45:35.553 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi105 2024-02-10T05:45:35.553 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T05:45:35.588 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085/remote/smithi110 2024-02-10T05:45:35.589 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-10T05:45:35.624 INFO:teuthology.task.internal:Removing archive directory... 2024-02-10T05:45:35.625 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T05:45:35.627 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T05:45:35.629 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-10T05:45:35.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-10T05:45:35.686 INFO:teuthology.task.internal:Not uploading archives. 2024-02-10T05:45:35.686 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-10T05:45:35.792 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-10T05:45:35.792 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T05:45:35.795 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T05:45:35.797 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-10T05:45:35.807 INFO:teuthology.orchestra.run.smithi029.stdout: 265694 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:45 /home/ubuntu/cephtest 2024-02-10T05:45:35.809 INFO:teuthology.orchestra.run.smithi105.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:45 /home/ubuntu/cephtest 2024-02-10T05:45:35.811 INFO:teuthology.orchestra.run.smithi110.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 10 05:45 /home/ubuntu/cephtest 2024-02-10T05:45:35.813 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-10T05:45:35.823 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-10T05:45:35.843 INFO:teuthology.nuke:Checking targets against current locks 2024-02-10T05:45:35.871 DEBUG:teuthology.nuke:shortname: smithi029 2024-02-10T05:45:35.871 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:45:35.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:21:33.274214', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFhCtJsG1Y7KtIGGRFYs/Qw4Gb3bJ/kDTOlam96Eze/TkWzv7QHpVzPR03nV5L/q69PJ+vRyTzPEGJpmeIVJPqk='} 2024-02-10T05:45:35.900 DEBUG:teuthology.nuke:shortname: smithi105 2024-02-10T05:45:35.900 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:45:35.930 DEBUG:teuthology.nuke:shortname: smithi110 2024-02-10T05:45:35.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-10T05:45:35.938 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:21:33.272747', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPyIf7TFhaSSvNFZNfHVTK2G3PwGZiyzwCnv2LgKRRQazhS5CvY3ki6HYcsuE4tQSRsVdPAA/pl1hA+ZU837MLE='} 2024-02-10T05:45:35.954 INFO:teuthology.orchestra.console.smithi029:Power off 2024-02-10T05:45:35.954 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T05:45:35.966 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:44:16-orch-main-distro-default-smithi/7555085', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-10 05:21:33.275459', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM55KO4V6S5wHn97KAIpq4FEXb4kwKujPIjWqsg/Rv4afevy8tyVREpmrfuCa+L3ZIhsV3gc15G9VjKPB6Qg6L0='} 2024-02-10T05:45:35.978 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-02-10T05:45:35.978 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:36.000 INFO:teuthology.orchestra.console.smithi105:Power off 2024-02-10T05:45:36.000 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T05:45:36.012 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-02-10T05:45:36.024 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2024-02-10T05:45:36.025 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:36.039 INFO:teuthology.orchestra.console.smithi110:Power off 2024-02-10T05:45:36.039 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-10T05:45:36.047 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-02-10T05:45:36.061 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2024-02-10T05:45:36.061 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:36.083 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-02-10T05:45:40.017 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:40.049 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:40.084 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:40.127 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-02-10T05:45:40.158 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-02-10T05:45:40.193 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2024-02-10T05:45:44.132 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:44.159 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:44.194 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-10T05:45:44.242 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-02-10T05:45:44.268 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-02-10T05:45:44.305 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2024-02-10T05:45:44.343 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-02-10T05:45:44.369 INFO:teuthology.orchestra.console.smithi105:Power off completed 2024-02-10T05:45:44.405 INFO:teuthology.orchestra.console.smithi110:Power off completed 2024-02-10T05:45:44.510 INFO:teuthology.run:Summary data: description: orch/cephadm/smoke-small/{0-distro/centos_9.stream_runc 0-nvme-loop agent/off fixed-2 mon_election/classic start} duration: 793.434939622879 failure_reason: '"2024-02-10T05:41:35.387173+0000 mon.a (mon.0) 191 : cluster 3 [WRN] MON_DOWN: 1/3 mons down, quorum a,c" in cluster log' owner: scheduled_yuriw@teuthology success: false 2024-02-10T05:45:44.510 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-10T05:45:44.584 INFO:teuthology.run:FAIL